Oct 30 03:56:16.543832 ip-10-0-139-99 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 03:56:16.979280 ip-10-0-139-99 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:56:16.979280 ip-10-0-139-99 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 03:56:16.979280 ip-10-0-139-99 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:56:16.979280 ip-10-0-139-99 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:56:16.983604 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983484    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:56:16.983604 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983603    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:56:16.983604 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983608    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983611    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983614    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983617    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983619    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983622    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983624    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983627    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983630    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983632    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983635    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983638    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983640    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983643    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983645    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983648    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983650    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983652    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983655    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:56:16.983695 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983658    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983661    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983663    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983665    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983669    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983671    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983674    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983676    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983686    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983689    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983692    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983695    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983698    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983700    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983703    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983706    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983709    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983711    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983714    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983716    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:56:16.984225 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983719    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983721    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983724    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983726    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983728    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983731    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983734    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983736    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983739    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983741    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983744    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983746    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983749    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983753    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983757    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983760    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983763    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983766    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983769    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:56:16.984753 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983771    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983774    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983776    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983779    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983782    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983784    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983787    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983789    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983792    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983794    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983797    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983799    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983801    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983804    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983806    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983809    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983811    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983814    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983816    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983819    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:56:16.985204 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983823    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983826    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983828    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983831    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983833    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983836    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983840    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983845    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983848    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983851    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983854    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983857    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983859    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983862    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983865    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983867    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983870    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983873    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983876    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:56:16.985709 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983878    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983881    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983883    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983885    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983888    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983890    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.983894    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985516    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985523    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985527    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985530    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985532    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985535    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985553    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985556    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985560    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985563    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985566    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985569    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985572    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:56:16.986143 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985576    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985578    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985581    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985583    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985586    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985588    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985591    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985595    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985599    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985602    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985605    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985608    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985610    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985613    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985615    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985618    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985620    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985623    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985625    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:56:16.986656 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985628    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985630    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985633    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985635    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985638    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985640    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985643    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985645    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985647    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985650    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985653    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985655    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985658    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985661    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985663    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985666    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985668    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985671    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985673    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:56:16.987116 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985676    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985679    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985681    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985683    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985686    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985689    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985692    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985694    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985697    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985699    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985701    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985705    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985708    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985711    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985713    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985716    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985718    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985721    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985724    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985727    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:56:16.987615 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985729    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985732    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985734    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985737    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985739    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985744    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985748    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985751    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985754    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985756    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985759    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985761    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985764    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985766    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985769    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985772    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985774    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985777    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985780    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:56:16.988092 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985782    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985785    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985787    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985790    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985793    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985795    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985797    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985800    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985802    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985805    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985807    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985809    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985812    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985814    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985817    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.985819    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986333    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986339    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986343    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986346    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:56:16.988556 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986348    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986351    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986354    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986356    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986359    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986362    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986365    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986367    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986370    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986372    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986375    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986377    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986381    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986385    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986388    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986390    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986393    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986396    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986398    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:56:16.989035 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986401    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986403    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986406    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986409    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986411    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986413    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986416    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986418    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986421    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986424    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986427    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986429    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986432    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986435    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986437    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986440    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986442    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986445    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986447    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:56:16.989478 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986449    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986452    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986454    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986457    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986459    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986462    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986464    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986467    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986469    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986471    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986474    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986477    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986479    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986481    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986484    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986487    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986490    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986492    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986495    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:56:16.989945 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986497    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986500    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986502    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986505    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986507    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986510    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986513    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986515    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986517    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986520    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986522    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986525    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986528    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986530    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986532    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986535    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986552    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986556    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986560    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986563    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:56:16.990389 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986566    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986569    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986572    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986574    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986577    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986579    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986582    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986586    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986590    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986593    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986595    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986598    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986601    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986603    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986607    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986609    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986612    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986614    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986617    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986620    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:56:16.990880 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986623    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986625    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986628    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986630    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.986633    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987720    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987730    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987736    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987741    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987745    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987748    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987752    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987757    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987760    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987763    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987767    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987770    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987773    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987776    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987779    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987783    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987786    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987789    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 03:56:16.991344 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987794    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987797    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987800    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987803    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987806    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987810    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987813    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987816    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987820    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987823    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987826    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987829    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987832    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987834    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987839    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987841    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987844    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987847    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987850    2577 flags.go:64] FLAG: --enable-server="true"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987853    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987857    2577 flags.go:64] FLAG: --event-burst="100"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987862    2577 flags.go:64] FLAG: --event-qps="50"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987866    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987869    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 03:56:16.991889 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987872    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987876    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987879    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987882    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987885    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987888    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987891    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987894    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987897    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987899    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987903    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987905    2577 flags.go:64] FLAG: --feature-gates=""
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987909    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987912    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987915    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987918    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987921    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987925    2577 flags.go:64] FLAG: --help="false"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987928    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-139-99.ec2.internal"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987931    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987934    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987937    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987940    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 03:56:16.992497 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987944    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987947    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987950    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987953    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987955    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987958    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987963    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987966    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987969    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987971    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987974    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987977    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987980    2577 flags.go:64] FLAG: --lock-file=""
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987983    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987986    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987989    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987995    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.987998    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988000    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988003    2577 flags.go:64] FLAG: --logging-format="text"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988006    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988010    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988013    2577 flags.go:64] FLAG: --manifest-url=""
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988015    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988020    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 03:56:16.993088 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988023    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988027    2577 flags.go:64] FLAG: --max-pods="110"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988031    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988034    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988036    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988039    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988042    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988045    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988048    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988055    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988058    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988061    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988064    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988068    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988071    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988074    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988077    2577 flags.go:64] FLAG: --port="10250"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988080    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988083    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08d9981b5d6272cfc"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988086    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988089    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988092    2577 flags.go:64] FLAG: --register-node="true"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988095    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988099    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988102    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988105    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 03:56:16.993667 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988108    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988112    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988115    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988118    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988120    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988124    2577 flags.go:64] FLAG: --runonce="false"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988126    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988131    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988134    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988138    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988140    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988143    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988146    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988149    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988152    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988155    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988158    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988161    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988164    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988168    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988171    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988176    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988179    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988182    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988186    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 30 03:56:16.994280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988189    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988191    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988194    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988197    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988200    2577 flags.go:64] FLAG: --v="2"
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988205    2577 flags.go:64] FLAG: --version="false"
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988209    2577 flags.go:64] FLAG: --vmodule=""
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988213    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988217    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988355    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988359    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988362    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988365    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988367    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988371    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988374    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988376    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988379    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988383    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988385    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988388    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988390    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:56:16.994911 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988393    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988396    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988398    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988401    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988405    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988407    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988410    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988412    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988415    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988417    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988420    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988422    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988425    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988427    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988430    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988432    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988435    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988437    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988440    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988442    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:56:16.995425 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988444    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988447    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988449    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988452    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988456    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988458    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988461    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988463    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988466    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988469    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988471    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988474    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988476    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988479    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988481    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988484    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988488    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988490    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988493    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988495    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:56:16.995912 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988498    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988500    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988502    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988505    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988507    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988510    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988513    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988515    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988518    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988520    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988522    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988525    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988528    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988530    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988533    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988535    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988554    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988559    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988563    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:56:16.996412 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988566    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988569    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988572    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988575    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988578    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988581    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988584    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988586    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988589    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988592    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988595    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988597    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988600    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988603    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988605    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988608    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988612    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988617    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988620    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:56:16.996861 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988623    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988626    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988629    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988631    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988634    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988636    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988640    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988644    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988646    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988649    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988653    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988656    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988658    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988661    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988664    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.988692    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:56:16.997306 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988734    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988739    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988743    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988746    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988748    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988751    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988753    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988757    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988760    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988762    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988765    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988767    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988770    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988772    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988775    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988778    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988780    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988783    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988786    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988788    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:56:16.997687 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988791    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988793    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988795    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988798    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988800    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988803    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988807    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988809    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988812    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988814    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988817    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988819    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988822    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988825    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988828    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988830    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988833    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988835    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988838    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988842    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:56:16.998149 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988845    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988847    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988849    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988852    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988854    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988857    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988859    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988862    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988864    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988867    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988869    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988873    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988876    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988879    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988881    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988883    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988886    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988888    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988892    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:56:16.998624 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988895    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988897    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988901    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988905    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988908    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988911    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988913    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988917    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988920    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988923    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988925    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988929    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988934    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988937    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988939    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988942    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988944    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988947    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988949    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:56:16.999078 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988952    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988954    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988957    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988959    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988961    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988964    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988966    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988969    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988971    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988974    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988976    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988978    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988981    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988985    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988988    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988990    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988992    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988996    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.988998    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.989000    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:56:16.999528 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.989003    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.989007    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.989009    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.989012    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.989014    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.989017    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.989020    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.989022    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.996245    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.996264    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.996280    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.996285    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996337    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996342    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996346    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996349    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996351    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996355    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996357    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:56:17.000049 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996360    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996362    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996365    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996367    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996370    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996372    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996375    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996378    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996380    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996383    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996385    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996388    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996391    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996394    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996396    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996399    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996401    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996404    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996406    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996409    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:56:17.000520 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996411    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996413    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996418    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996420    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996423    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996426    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996428    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996431    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996433    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996435    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996438    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996440    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996443    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996445    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996447    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996450    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996452    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996455    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996457    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996460    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:56:17.001073 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996462    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996465    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996467    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996469    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996472    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996474    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996477    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996479    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996483    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996486    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996489    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996492    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996494    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996498    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996501    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996505    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996507    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996510    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996512    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:56:17.001578 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996515    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996517    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996520    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996522    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996525    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996527    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996529    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996532    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996534    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996554    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996561    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996565    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996568    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996571    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996573    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996576    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996578    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996581    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996583    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:56:17.002021 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996586    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996588    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996591    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996593    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996596    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996598    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996601    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996604    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996606    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996609    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996612    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996615    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996617    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996620    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996622    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996625    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996627    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996629    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996632    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996634    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:56:17.002451 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996637    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.996663    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996798    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996802    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996805    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996807    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996810    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996812    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996816    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996820    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996823    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996826    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996829    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996832    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996834    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:56:17.002959 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996837    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996839    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996842    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996844    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996847    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996850    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996852    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996855    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996858    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996860    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996863    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996866    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996868    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996871    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996875    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996878    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996881    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996884    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996887    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:56:17.003310 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996889    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996891    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996894    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996896    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996898    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996901    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996903    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996906    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996908    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996911    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996913    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996916    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996918    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996920    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996923    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996925    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996928    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996931    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996933    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996936    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:56:17.003762 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996938    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996940    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996943    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996946    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996948    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996951    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996953    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996955    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996958    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996960    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996963    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996965    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996968    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996970    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996973    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996975    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996977    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996980    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996982    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996985    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:56:17.004221 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996987    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996990    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996992    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996994    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996997    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.996999    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997002    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997004    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997007    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997009    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997012    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997014    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997016    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997019    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997021    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997024    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997027    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997029    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997032    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997034    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:56:17.004700 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997036    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997038    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997041    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997043    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997045    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997048    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997055    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997057    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997060    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997062    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997064    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997067    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997069    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997072    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.997094    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997124    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997128    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:56:17.005152 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997131    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997134    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997137    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997139    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997142    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997144    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997147    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997149    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997152    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997155    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997157    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997160    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997163    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997166    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997169    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997173    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997175    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997178    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997182    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:56:17.005597 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997184    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997187    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997190    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997194    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997197    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997199    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997202    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997204    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997207    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997209    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997212    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997215    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997217    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997220    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997222    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997225    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997228    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997232    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997234    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:56:17.006042 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997237    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997239    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997242    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997244    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997247    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997249    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997252    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997255    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997258    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997260    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997263    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997266    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997268    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997271    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997273    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997275    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997279    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997282    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997285    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:56:17.006471 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997287    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997289    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997292    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997294    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997297    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997299    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997301    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997304    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997306    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997309    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997311    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997314    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997317    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997320    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997322    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997325    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997327    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997330    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997332    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997335    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:56:17.006933 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997337    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997340    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997342    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997345    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997348    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997350    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997353    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997355    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997359    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997363    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997367    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997370    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997372    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997374    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997377    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997380    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997382    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997384    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997387    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:56:17.007508 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997389    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997392    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997394    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997397    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997399    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997402    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997404    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:16.997407    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.997461    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:16.999746    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.000794    2577 server.go:1014] "Starting client certificate rotation"
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.000893    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.000927    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 03:56:17.008270 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.002983    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:56:17.009302 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.009282    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:56:17.017170 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.017153    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 30 03:56:17.020708 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.020692    2577 log.go:25] "Validated CRI v1 image API"
Oct 30 03:56:17.021592 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.021566    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 03:56:17.024261 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.024241    2577 fs.go:135] Filesystem UUIDs: map[40fa5620-ebf4-4db7-af42-980fddacc43a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f8891625-04a4-4fa0-925c-b36013e04348:/dev/nvme0n1p4]
Oct 30 03:56:17.024310 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.024260    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 03:56:17.029314 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.029208    2577 manager.go:217] Machine: {Timestamp:2026-10-30 03:56:17.027809138 +0000 UTC m=+0.370568768 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099919 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21eca4969f6e2b5bed4d0fcc570234 SystemUUID:ec21eca4-969f-6e2b-5bed-4d0fcc570234 BootID:fd54cb52-50c0-4f5b-b1d8-116a83ab07ae Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:91:05:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:91:05:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:d0:ef:4b:af:25 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 03:56:17.029314 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.029313    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 03:56:17.029425 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.029388    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 03:56:17.032632 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.032605    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 03:56:17.032758 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.032632    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 03:56:17.032806 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.032761    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 03:56:17.032806 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.032769    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 03:56:17.032806 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.032779    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 03:56:17.033515 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.033505    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 03:56:17.034662 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.034653    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 03:56:17.038836 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.038824    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 03:56:17.038959 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.038950    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 03:56:17.039777 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.039767    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 03:56:17.039835 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.039781    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 03:56:17.039835 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.039794    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 03:56:17.039835 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.039807    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 30 03:56:17.039835 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.039815    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 03:56:17.040117 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.040103    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 03:56:17.042820 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.042803    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 03:56:17.050406 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.050393    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 03:56:17.051338 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.051322    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 03:56:17.051416 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.051352    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 03:56:17.051556 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.051530    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 03:56:17.051601 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.051566    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 03:56:17.051601 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.051574    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 03:56:17.051601 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.051580    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 03:56:17.051601 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.051599    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 03:56:17.051813 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.051605    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 03:56:17.052202 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.052192    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 03:56:17.052243 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.052202    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 03:56:17.052243 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.052209    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 03:56:17.052243 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.052215    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 03:56:17.052243 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.052230    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 03:56:17.052243 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.052239    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 03:56:17.053140 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.053130    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 03:56:17.053140 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.053139    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 03:56:17.057033 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.057019    2577 server.go:1263] "Started kubelet"
Oct 30 03:56:17.057112 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.057087    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 03:56:17.057163 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.057117    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 03:56:17.057207 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.057183    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 03:56:17.061977 ip-10-0-139-99 systemd[1]: Started Kubernetes Kubelet.
Oct 30 03:56:17.062994 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.062879    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 03:56:17.064638 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.064622    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 03:56:17.067779 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.067757    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 03:56:17.071665 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.071649    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 03:56:17.072202 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.072190    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 03:56:17.072295 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.072283    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:17.072949 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.072931    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 03:56:17.073007 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.072950    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 03:56:17.073007 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.072953    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 03:56:17.073071 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.073031    2577 factory.go:55] Registering systemd factory
Oct 30 03:56:17.073071 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.073048    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 30 03:56:17.073071 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.073051    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 03:56:17.073071 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.073057    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 03:56:17.073280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.073262    2577 factory.go:153] Registering CRI-O factory
Oct 30 03:56:17.073280 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.073276    2577 factory.go:223] Registration of the crio container factory successfully
Oct 30 03:56:17.073399 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.073313    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 03:56:17.073399 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.073327    2577 factory.go:103] Registering Raw factory
Oct 30 03:56:17.073399 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.073337    2577 manager.go:1199] Started watching for new ooms in manager
Oct 30 03:56:17.074354 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.074336    2577 manager.go:319] Starting recovery of all containers
Oct 30 03:56:17.084657 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.084641    2577 manager.go:324] Recovery completed
Oct 30 03:56:17.086565 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.086531    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 03:56:17.089258 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.089246    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:56:17.090939 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.090920    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:56:17.091018 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.090954    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:56:17.091018 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.090969    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:56:17.091554 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.091522    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 30 03:56:17.091616 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.091553    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 03:56:17.091616 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.091574    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 03:56:17.091616 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.091536    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:56:17.092231 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.092214    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:56:17.092292 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.092242    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:56:17.092292 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.092251    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:56:17.093791 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.093773    2577 policy_none.go:50] "Start"
Oct 30 03:56:17.093791 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.093791    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 03:56:17.093903 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.093801    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 03:56:17.095783 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.095771    2577 policy_none.go:44] "Start"
Oct 30 03:56:17.108226 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.108103    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 03:56:17.108341 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.108320    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 03:56:17.109489 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.108303    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-99.ec2.internal.18e3327a339e45eb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-99.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-99.ec2.internal,},FirstTimestamp:2026-10-30 03:56:17.056998891 +0000 UTC m=+0.399758522,LastTimestamp:2026-10-30 03:56:17.056998891 +0000 UTC m=+0.399758522,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-99.ec2.internal,}"
Oct 30 03:56:17.110209 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.110190    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 03:56:17.118091 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.118061    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 03:56:17.118245 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.118127    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 03:56:17.123359 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.123292    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-99.ec2.internal.18e3327a35a422f8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-99.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-99.ec2.internal,},FirstTimestamp:2026-10-30 03:56:17.090937592 +0000 UTC m=+0.433697241,LastTimestamp:2026-10-30 03:56:17.090937592 +0000 UTC m=+0.433697241,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-99.ec2.internal,}"
Oct 30 03:56:17.123441 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.123424    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 03:56:17.128910 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.128889    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-565cn"
Oct 30 03:56:17.142302 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.142272    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.144597    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-99.ec2.internal.18e3327a35a47bd3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-99.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-99.ec2.internal,},FirstTimestamp:2026-10-30 03:56:17.090960339 +0000 UTC m=+0.433719970,LastTimestamp:2026-10-30 03:56:17.090960339 +0000 UTC m=+0.433719970,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-99.ec2.internal,}"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.144903    2577 manager.go:341] "Starting Device Plugin manager"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.144948    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.144960    2577 server.go:87] "Starting device plugin registration server"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.145123    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-565cn"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.145186    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.145197    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.145328    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.145411    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.145418    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.145846    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.145888    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.155688    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.156935    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.156955    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.156970    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 03:56:17.159393 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.156998    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 03:56:17.162849 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.162692    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-99.ec2.internal.18e3327a35a4b2be  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-99.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-99.ec2.internal,},FirstTimestamp:2026-10-30 03:56:17.090974398 +0000 UTC m=+0.433734035,LastTimestamp:2026-10-30 03:56:17.090974398 +0000 UTC m=+0.433734035,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-99.ec2.internal,}"
Oct 30 03:56:17.203617 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.203595    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:56:17.246125 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.246076    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:56:17.247447 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.247432    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:56:17.247512 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.247457    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:56:17.247512 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.247467    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:56:17.247512 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.247490    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.257341 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.257318    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"]
Oct 30 03:56:17.257393 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.257377    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:56:17.258111 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.258099    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:56:17.258172 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.258124    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:56:17.258172 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.258135    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:56:17.260343 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.260331    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:56:17.260510 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.260476    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.260582 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.260534    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:56:17.261081 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.261067    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:56:17.261147 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.261096    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:56:17.261147 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.261107    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:56:17.261205 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.261067    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:56:17.261205 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.261169    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:56:17.261205 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.261191    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:56:17.263166 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.263152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.263271 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.263174    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:56:17.263816 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.263801    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:56:17.263901 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.263821    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:56:17.263901 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.263830    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:56:17.265841 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.265828    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.265899 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.265847    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-99.ec2.internal\": node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:17.291209 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.291191    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-99.ec2.internal\" not found" node="ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.293744 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.293729    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-99.ec2.internal\" not found" node="ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.312579 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.312560    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:17.374471 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.374449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be906c4a667f460e0e8153479b6ef733-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"be906c4a667f460e0e8153479b6ef733\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.374568 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.374475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be906c4a667f460e0e8153479b6ef733-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"be906c4a667f460e0e8153479b6ef733\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.374568 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.374497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f95db1d51f36d0a32fb67f82ccd47c76-config\") pod \"kube-apiserver-proxy-ip-10-0-139-99.ec2.internal\" (UID: \"f95db1d51f36d0a32fb67f82ccd47c76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.412669 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.412650    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:17.475108 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.475086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be906c4a667f460e0e8153479b6ef733-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"be906c4a667f460e0e8153479b6ef733\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.475197 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.475111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be906c4a667f460e0e8153479b6ef733-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"be906c4a667f460e0e8153479b6ef733\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.475197 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.475132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f95db1d51f36d0a32fb67f82ccd47c76-config\") pod \"kube-apiserver-proxy-ip-10-0-139-99.ec2.internal\" (UID: \"f95db1d51f36d0a32fb67f82ccd47c76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.475293 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.475196    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f95db1d51f36d0a32fb67f82ccd47c76-config\") pod \"kube-apiserver-proxy-ip-10-0-139-99.ec2.internal\" (UID: \"f95db1d51f36d0a32fb67f82ccd47c76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.475293 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.475205    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be906c4a667f460e0e8153479b6ef733-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"be906c4a667f460e0e8153479b6ef733\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.475293 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.475209    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be906c4a667f460e0e8153479b6ef733-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"be906c4a667f460e0e8153479b6ef733\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.513439 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.513387    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:17.596340 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.596314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.598498 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:17.598480    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:17.614354 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.614334    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:17.714946 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.714905    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:17.815408 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.815339    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:17.915919 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:17.915894    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:18.000396 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.000371    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 03:56:18.000877 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.000558    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 03:56:18.016685 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:18.016659    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:18.117168 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:18.117105    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:18.151386 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.151358    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 03:51:18 +0000 UTC" deadline="2028-06-18 07:20:04.745017553 +0000 UTC"
Oct 30 03:56:18.151386 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.151386    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14331h23m46.593634772s"
Oct 30 03:56:18.166280 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:18.166251    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95db1d51f36d0a32fb67f82ccd47c76.slice/crio-71b0828b77c7b2bc8ef6dae22ed644fe424ae384c7d887b6879527b1c4e121f5 WatchSource:0}: Error finding container 71b0828b77c7b2bc8ef6dae22ed644fe424ae384c7d887b6879527b1c4e121f5: Status 404 returned error can't find the container with id 71b0828b77c7b2bc8ef6dae22ed644fe424ae384c7d887b6879527b1c4e121f5
Oct 30 03:56:18.166904 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:18.166882    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe906c4a667f460e0e8153479b6ef733.slice/crio-71809b6f9c61f215314c5d4a89b2bd5059bbaf724679fa389a0239400a4e0e41 WatchSource:0}: Error finding container 71809b6f9c61f215314c5d4a89b2bd5059bbaf724679fa389a0239400a4e0e41: Status 404 returned error can't find the container with id 71809b6f9c61f215314c5d4a89b2bd5059bbaf724679fa389a0239400a4e0e41
Oct 30 03:56:18.171454 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.171435    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:56:18.188666 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.187316    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:56:18.198058 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.198041    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:56:18.199517 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.199501    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:56:18.199620 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.199531    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:56:18.199620 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.199564    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:56:18.217667 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:18.217644    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:18.318359 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:18.318328    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:18.418853 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:18.418786    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:56:18.480470 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.480442    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:56:18.572974 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.572944    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:18.600148 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.600122    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:56:18.601325 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.601303    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Oct 30 03:56:18.626963 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.626937    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:56:18.627152 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:18.627138    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:56:20.554822 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.554743    2577 apiserver.go:52] "Watching apiserver"
Oct 30 03:56:20.585617 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.585582    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 03:56:20.599761 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.599733    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 03:56:20.600090 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.600056    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-d7nsr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn","openshift-dns/node-resolver-6njd4","openshift-multus/multus-additional-cni-plugins-2gkl8","openshift-ovn-kubernetes/ovnkube-node-c6hqf","kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qz4zz","openshift-network-operator/iptables-alerter-6czxj","openshift-multus/network-metrics-daemon-b67qk","openshift-network-diagnostics/network-check-target-k2wgk","kube-system/global-pull-secret-syncer-d67gd","openshift-image-registry/node-ca-mc77t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal","openshift-multus/multus-ttzr9"]
Oct 30 03:56:20.602675 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.602649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:20.602788 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:20.602742    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:20.604795 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.604773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.607008 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.606991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:56:20.609438 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.609420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:56:20.611556 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.611535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.614185 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.614159    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:56:20.616810 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.616794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.619144 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.619128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:56:20.621060 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.621034    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:20.621224 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:20.621189    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:20.621665 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.621647    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 03:56:20.621784 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.621771    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h7p9v\""
Oct 30 03:56:20.621846 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.621822    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 03:56:20.621946 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.621917    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 03:56:20.622855 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.622836    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vm8lh\""
Oct 30 03:56:20.622943 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.622876    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 03:56:20.622943 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.622884    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 03:56:20.623374 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.623357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.625222 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.625174    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 03:56:20.626189 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.626167    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:20.626279 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:20.626254    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:20.628626 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.628602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.640380 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.640357    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 03:56:20.640478 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.640436    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9pfq2"
Oct 30 03:56:20.640478 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.640463    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9pfq2"
Oct 30 03:56:20.674934 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.674662    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal" event={"ID":"f95db1d51f36d0a32fb67f82ccd47c76","Type":"ContainerStarted","Data":"71b0828b77c7b2bc8ef6dae22ed644fe424ae384c7d887b6879527b1c4e121f5"}
Oct 30 03:56:20.676054 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.676017    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal" event={"ID":"be906c4a667f460e0e8153479b6ef733","Type":"ContainerStarted","Data":"71809b6f9c61f215314c5d4a89b2bd5059bbaf724679fa389a0239400a4e0e41"}
Oct 30 03:56:20.687554 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.687536    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 03:56:20.698419 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysconfig\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.698500 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-systemd\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.698500 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-var-lib-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.698603 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.698603 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-env-overrides\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.698663 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovn-node-metrics-cert\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.698663 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-device-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.698663 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-netns\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.698797 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78q6r\" (UniqueName: \"kubernetes.io/projected/99e3f70c-6dd8-467e-a444-7692e30a76a4-kube-api-access-78q6r\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.698797 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.698797 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.698797 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc9a426a-587d-4611-9298-90fc4b4f1dc3-dbus\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:20.698969 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41551065-0510-4289-a742-83987e916774-iptables-alerter-script\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:56:20.698969 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntgd\" (UniqueName: \"kubernetes.io/projected/41551065-0510-4289-a742-83987e916774-kube-api-access-4ntgd\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:56:20.698969 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-kubelet\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.698969 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-log-socket\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.698969 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.698960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovnkube-config\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.699149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-socket-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.699149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.699149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.699149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b555cf82-5423-4c47-a004-ae399654683b-tmp-dir\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:56:20.699149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-kubernetes\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.699149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99e3f70c-6dd8-467e-a444-7692e30a76a4-tmp\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.699368 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:20.699368 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nfhp\" (UniqueName: \"kubernetes.io/projected/b555cf82-5423-4c47-a004-ae399654683b-kube-api-access-5nfhp\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:56:20.699368 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-lib-modules\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.699368 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-systemd-units\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.699368 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-run-netns\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.699368 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-ovn\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.699368 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.699368 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6fec406-0515-44b5-ae95-c2fbfe84b452-cni-binary-copy\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.699368 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysctl-d\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.699368 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t2hj\" (UniqueName: \"kubernetes.io/projected/623e4b93-06e4-44f9-bd85-d9b663df4834-kube-api-access-6t2hj\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-system-cni-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-daemon-config\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-cni-netd\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-cnibin\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-cnibin\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-slash\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4n7t\" (UniqueName: \"kubernetes.io/projected/cbf6ac89-eb35-4d19-a28c-6f879f02703a-kube-api-access-h4n7t\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr8xt\" (UniqueName: \"kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt\") pod \"network-check-target-k2wgk\" (UID: \"4c692538-d934-46cf-8c4c-d6a888dd00a1\") " pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-cni-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-cni-multus\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0384e218-1937-4099-8566-67775cda0521-host\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-node-log\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.699645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-run-ovn-kubernetes\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-sys-fs\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-etc-kubernetes\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-host\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-etc-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-cni-bin\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-system-cni-dir\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-hostroot\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcl9n\" (UniqueName: \"kubernetes.io/projected/c6fec406-0515-44b5-ae95-c2fbfe84b452-kube-api-access-qcl9n\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-run\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e7b0fed-186b-4d7b-affa-e91d5ef4b523-konnectivity-ca\") pod \"konnectivity-agent-d7nsr\" (UID: \"8e7b0fed-186b-4d7b-affa-e91d5ef4b523\") " pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc9a426a-587d-4611-9298-90fc4b4f1dc3-kubelet-config\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-multus-certs\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41551065-0510-4289-a742-83987e916774-host-slash\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-tuned\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.699988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-systemd\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.700110 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-registration-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b555cf82-5423-4c47-a004-ae399654683b-hosts-file\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700110    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-socket-dir-parent\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-k8s-cni-cncf-io\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-kubelet\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysctl-conf\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-sys\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpt9x\" (UniqueName: \"kubernetes.io/projected/0384e218-1937-4099-8566-67775cda0521-kube-api-access-lpt9x\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-295lb\" (UniqueName: \"kubernetes.io/projected/478b6cf0-d0dd-4d65-aafb-ff211b60c188-kube-api-access-295lb\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e7b0fed-186b-4d7b-affa-e91d5ef4b523-agent-certs\") pod \"konnectivity-agent-d7nsr\" (UID: \"8e7b0fed-186b-4d7b-affa-e91d5ef4b523\") " pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-cni-bin\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-conf-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0384e218-1937-4099-8566-67775cda0521-serviceca\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:56:20.700642 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-modprobe-d\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.701160 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-var-lib-kubelet\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.701160 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovnkube-script-lib\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.701160 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.700565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mr28\" (UniqueName: \"kubernetes.io/projected/9e6da480-8072-42d2-98f3-91566744aa96-kube-api-access-5mr28\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.722377 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.722355    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 03:56:20.722573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.722551    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 03:56:20.722684 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.722589    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 03:56:20.722684 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.722659    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 03:56:20.722761 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.722677    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 03:56:20.722761 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.722704    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kwf82\""
Oct 30 03:56:20.722761 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.722735    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 03:56:20.722761 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.722736    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 03:56:20.722916 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.722761    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 03:56:20.722966 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.722936    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:56:20.723011 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.722990    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 03:56:20.723011 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.723002    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 03:56:20.723109 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.723012    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 03:56:20.723109 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.723030    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 03:56:20.723109 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.723089    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 03:56:20.723109 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.723108    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-smvt7\""
Oct 30 03:56:20.723298 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.723166    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4xznk\""
Oct 30 03:56:20.723298 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.723244    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 03:56:20.724410 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.724386    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:56:20.724410 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.724395    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 03:56:20.724540 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.724456    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 03:56:20.724540 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.724460    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rs54w\""
Oct 30 03:56:20.724540 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.724494    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 03:56:20.736617 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.736598    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-skmdf\""
Oct 30 03:56:20.736696 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.736625    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dnvhh\""
Oct 30 03:56:20.736696 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.736683    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 03:56:20.736786 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.736744    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 03:56:20.736786 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.736782    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w4blj\""
Oct 30 03:56:20.800781 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.800750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-systemd\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.800905 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.800795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-var-lib-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.800905 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.800821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.800905 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.800848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-env-overrides\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.800905 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.800864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovn-node-metrics-cert\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.800905 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.800883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-device-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.800905 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.800901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-netns\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.801149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.800920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78q6r\" (UniqueName: \"kubernetes.io/projected/99e3f70c-6dd8-467e-a444-7692e30a76a4-kube-api-access-78q6r\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.801149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.800939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.801149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.800955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.801149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.800968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc9a426a-587d-4611-9298-90fc4b4f1dc3-dbus\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:20.801149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.800985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41551065-0510-4289-a742-83987e916774-iptables-alerter-script\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:56:20.801149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntgd\" (UniqueName: \"kubernetes.io/projected/41551065-0510-4289-a742-83987e916774-kube-api-access-4ntgd\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:56:20.801149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-kubelet\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.801149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-log-socket\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.801149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovnkube-config\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.801149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-socket-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.801149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.801149 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.801684 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b555cf82-5423-4c47-a004-ae399654683b-tmp-dir\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:56:20.801684 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-kubernetes\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.801684 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99e3f70c-6dd8-467e-a444-7692e30a76a4-tmp\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.801684 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:20.801684 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801236    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-var-lib-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.801684 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801449    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.801684 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801563    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.802028 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801987    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-env-overrides\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.802105 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.802072    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.802330 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.802312    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc9a426a-587d-4611-9298-90fc4b4f1dc3-dbus\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:20.802407 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.802338    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 03:56:20.802407 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.802366    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-device-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.802503 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.802491    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.802721 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.802701    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41551065-0510-4289-a742-83987e916774-iptables-alerter-script\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:56:20.802795 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801151    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-netns\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.802945 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nfhp\" (UniqueName: \"kubernetes.io/projected/b555cf82-5423-4c47-a004-ae399654683b-kube-api-access-5nfhp\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:56:20.802999 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.802967    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b555cf82-5423-4c47-a004-ae399654683b-tmp-dir\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:56:20.802999 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.802972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-lib-modules\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.803109 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.802997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-systemd-units\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.803109 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-run-netns\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.803109 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-ovn\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.803109 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.803109 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6fec406-0515-44b5-ae95-c2fbfe84b452-cni-binary-copy\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.803335 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysctl-d\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.803335 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t2hj\" (UniqueName: \"kubernetes.io/projected/623e4b93-06e4-44f9-bd85-d9b663df4834-kube-api-access-6t2hj\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:20.803335 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-system-cni-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.803335 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-daemon-config\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.803335 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-cni-netd\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.803335 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-cnibin\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.803573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803366    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-socket-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.803573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-cnibin\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.803573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.803573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.801202    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-systemd\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.803573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-slash\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.803573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803490    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovnkube-config\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.803573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4n7t\" (UniqueName: \"kubernetes.io/projected/cbf6ac89-eb35-4d19-a28c-6f879f02703a-kube-api-access-h4n7t\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.803573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8xt\" (UniqueName: \"kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt\") pod \"network-check-target-k2wgk\" (UID: \"4c692538-d934-46cf-8c4c-d6a888dd00a1\") " pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:20.803573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803553    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-kubernetes\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.803573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-cni-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-cni-multus\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803616    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-cni-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0384e218-1937-4099-8566-67775cda0521-host\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803678    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-kubelet\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803720    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-log-socket\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803759    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-cni-multus\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0384e218-1937-4099-8566-67775cda0521-host\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-node-log\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-run-ovn-kubernetes\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803854    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-system-cni-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803859    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6fec406-0515-44b5-ae95-c2fbfe84b452-cni-binary-copy\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-sys-fs\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-etc-kubernetes\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.803924 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-host\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-etc-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-cni-bin\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803984    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-lib-modules\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.803991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-system-cni-dir\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-hostroot\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804022    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-systemd-units\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcl9n\" (UniqueName: \"kubernetes.io/projected/c6fec406-0515-44b5-ae95-c2fbfe84b452-kube-api-access-qcl9n\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804047    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-run-netns\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-run\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804088    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-ovn\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e7b0fed-186b-4d7b-affa-e91d5ef4b523-konnectivity-ca\") pod \"konnectivity-agent-d7nsr\" (UID: \"8e7b0fed-186b-4d7b-affa-e91d5ef4b523\") " pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc9a426a-587d-4611-9298-90fc4b4f1dc3-kubelet-config\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-multus-certs\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41551065-0510-4289-a742-83987e916774-host-slash\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-tuned\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-systemd\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.804426 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-registration-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b555cf82-5423-4c47-a004-ae399654683b-hosts-file\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-socket-dir-parent\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804336    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-daemon-config\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-k8s-cni-cncf-io\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-kubelet\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysctl-conf\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804435    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-node-log\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-sys\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804480    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-run-ovn-kubernetes\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804499    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpt9x\" (UniqueName: \"kubernetes.io/projected/0384e218-1937-4099-8566-67775cda0521-kube-api-access-lpt9x\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804533    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804555    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-run\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804395    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-k8s-cni-cncf-io\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.805046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-295lb\" (UniqueName: \"kubernetes.io/projected/478b6cf0-d0dd-4d65-aafb-ff211b60c188-kube-api-access-295lb\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804697    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysctl-d\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804734    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-system-cni-dir\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804761    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc9a426a-587d-4611-9298-90fc4b4f1dc3-kubelet-config\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804775    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-cni-netd\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804812    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-slash\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.804973    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-cnibin\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805025    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-cnibin\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805068    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e7b0fed-186b-4d7b-affa-e91d5ef4b523-konnectivity-ca\") pod \"konnectivity-agent-d7nsr\" (UID: \"8e7b0fed-186b-4d7b-affa-e91d5ef4b523\") " pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805137    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-hostroot\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805186    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-host\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805223    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-etc-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805260    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-cni-bin\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:20.805336    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805368    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysctl-conf\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:20.805395    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret podName:bc9a426a-587d-4611-9298-90fc4b4f1dc3 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:21.30537662 +0000 UTC m=+3.134600970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret") pod "global-pull-secret-syncer-d67gd" (UID: "bc9a426a-587d-4611-9298-90fc4b4f1dc3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805412    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-kubelet\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.805753 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e7b0fed-186b-4d7b-affa-e91d5ef4b523-agent-certs\") pod \"konnectivity-agent-d7nsr\" (UID: \"8e7b0fed-186b-4d7b-affa-e91d5ef4b523\") " pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-cni-bin\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805580    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b555cf82-5423-4c47-a004-ae399654683b-hosts-file\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805589    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-conf-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0384e218-1937-4099-8566-67775cda0521-serviceca\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805639    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-etc-kubernetes\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805696    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-socket-dir-parent\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-modprobe-d\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-var-lib-kubelet\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:20.805773    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:20.805851    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs podName:623e4b93-06e4-44f9-bd85-d9b663df4834 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:21.305832488 +0000 UTC m=+3.135056834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs") pod "network-metrics-daemon-b67qk" (UID: "623e4b93-06e4-44f9-bd85-d9b663df4834") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805900    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-multus-certs\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805908    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-sys-fs\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805946    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-var-lib-kubelet\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.806020    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-modprobe-d\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.806573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.806069    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-conf-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.807605 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.806134    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-systemd\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.807605 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.806175    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0384e218-1937-4099-8566-67775cda0521-serviceca\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:56:20.807605 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.806187    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-sys\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.807605 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.806218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovnkube-script-lib\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.807605 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.806263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mr28\" (UniqueName: \"kubernetes.io/projected/9e6da480-8072-42d2-98f3-91566744aa96-kube-api-access-5mr28\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.807605 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.806309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysconfig\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.807605 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.805859    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-cni-bin\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.807605 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.806392    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-registration-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.807605 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.806460    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysconfig\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.807605 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.806484    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41551065-0510-4289-a742-83987e916774-host-slash\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:56:20.807605 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.807387    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovn-node-metrics-cert\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.808095 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.808057    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99e3f70c-6dd8-467e-a444-7692e30a76a4-tmp\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.808520 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.808490    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovnkube-script-lib\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.809245 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.809224    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-tuned\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.809467 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.809446    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e7b0fed-186b-4d7b-affa-e91d5ef4b523-agent-certs\") pod \"konnectivity-agent-d7nsr\" (UID: \"8e7b0fed-186b-4d7b-affa-e91d5ef4b523\") " pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:56:20.826211 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.826159    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78q6r\" (UniqueName: \"kubernetes.io/projected/99e3f70c-6dd8-467e-a444-7692e30a76a4-kube-api-access-78q6r\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.826387 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.826366    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nfhp\" (UniqueName: \"kubernetes.io/projected/b555cf82-5423-4c47-a004-ae399654683b-kube-api-access-5nfhp\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:56:20.826835 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.826813    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntgd\" (UniqueName: \"kubernetes.io/projected/41551065-0510-4289-a742-83987e916774-kube-api-access-4ntgd\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:56:20.827930 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.827910    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t2hj\" (UniqueName: \"kubernetes.io/projected/623e4b93-06e4-44f9-bd85-d9b663df4834-kube-api-access-6t2hj\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:20.828476 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:20.828458    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:56:20.828476 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:20.828478    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:56:20.828619 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:20.828491    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cr8xt for pod openshift-network-diagnostics/network-check-target-k2wgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:56:20.828619 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:20.828546    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt podName:4c692538-d934-46cf-8c4c-d6a888dd00a1 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:21.328530065 +0000 UTC m=+3.157754394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cr8xt" (UniqueName: "kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt") pod "network-check-target-k2wgk" (UID: "4c692538-d934-46cf-8c4c-d6a888dd00a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:56:20.831158 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.831067    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcl9n\" (UniqueName: \"kubernetes.io/projected/c6fec406-0515-44b5-ae95-c2fbfe84b452-kube-api-access-qcl9n\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.831383 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.831342    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4n7t\" (UniqueName: \"kubernetes.io/projected/cbf6ac89-eb35-4d19-a28c-6f879f02703a-kube-api-access-h4n7t\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:20.841215 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.841187    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpt9x\" (UniqueName: \"kubernetes.io/projected/0384e218-1937-4099-8566-67775cda0521-kube-api-access-lpt9x\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:56:20.843747 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.843687    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mr28\" (UniqueName: \"kubernetes.io/projected/9e6da480-8072-42d2-98f3-91566744aa96-kube-api-access-5mr28\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.846443 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.846416    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-295lb\" (UniqueName: \"kubernetes.io/projected/478b6cf0-d0dd-4d65-aafb-ff211b60c188-kube-api-access-295lb\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.883706 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.883677    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:56:20.925269 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.925247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:56:20.927845 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.927783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:56:20.937629 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.937580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:56:20.940482 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:20.940454    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478b6cf0_d0dd_4d65_aafb_ff211b60c188.slice/crio-2353c7b074a790f5e9c847cbc30bb3566ed8cdb8aafa871553776ea0972f8f75 WatchSource:0}: Error finding container 2353c7b074a790f5e9c847cbc30bb3566ed8cdb8aafa871553776ea0972f8f75: Status 404 returned error can't find the container with id 2353c7b074a790f5e9c847cbc30bb3566ed8cdb8aafa871553776ea0972f8f75
Oct 30 03:56:20.942452 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.942392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ttzr9"
Oct 30 03:56:20.950446 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:20.950353    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7b0fed_186b_4d7b_affa_e91d5ef4b523.slice/crio-de458eaf8dd33fa9b7e09166d8cfdf13201dbdece842f3396a4b802ec4da6d8b WatchSource:0}: Error finding container de458eaf8dd33fa9b7e09166d8cfdf13201dbdece842f3396a4b802ec4da6d8b: Status 404 returned error can't find the container with id de458eaf8dd33fa9b7e09166d8cfdf13201dbdece842f3396a4b802ec4da6d8b
Oct 30 03:56:20.953359 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:20.953167    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fec406_0515_44b5_ae95_c2fbfe84b452.slice/crio-18eed020933973ad437dd0e5595f20105d27486ffa38d4da81dabf0cec261f3b WatchSource:0}: Error finding container 18eed020933973ad437dd0e5595f20105d27486ffa38d4da81dabf0cec261f3b: Status 404 returned error can't find the container with id 18eed020933973ad437dd0e5595f20105d27486ffa38d4da81dabf0cec261f3b
Oct 30 03:56:20.953359 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.953197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:56:20.963737 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:20.963711    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0384e218_1937_4099_8566_67775cda0521.slice/crio-3f54ad826dc14af64a98e5eef31e80bd30ee7dd25712a14a85bad38fad43b133 WatchSource:0}: Error finding container 3f54ad826dc14af64a98e5eef31e80bd30ee7dd25712a14a85bad38fad43b133: Status 404 returned error can't find the container with id 3f54ad826dc14af64a98e5eef31e80bd30ee7dd25712a14a85bad38fad43b133
Oct 30 03:56:20.973882 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.973855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:56:20.976041 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.976019    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:56:20.977439 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.977421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:56:20.985598 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:20.985570    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6da480_8072_42d2_98f3_91566744aa96.slice/crio-89472633618f57002f7eef82f103ad7c8616a1475fe681f81d5a48004b399602 WatchSource:0}: Error finding container 89472633618f57002f7eef82f103ad7c8616a1475fe681f81d5a48004b399602: Status 404 returned error can't find the container with id 89472633618f57002f7eef82f103ad7c8616a1475fe681f81d5a48004b399602
Oct 30 03:56:20.989885 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:20.989852    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41551065_0510_4289_a742_83987e916774.slice/crio-e3f807feb80a7f1d40ed6617d546ee2855991902856e0a2a30ee79d0325386c7 WatchSource:0}: Error finding container e3f807feb80a7f1d40ed6617d546ee2855991902856e0a2a30ee79d0325386c7: Status 404 returned error can't find the container with id e3f807feb80a7f1d40ed6617d546ee2855991902856e0a2a30ee79d0325386c7
Oct 30 03:56:20.998408 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:20.998385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:21.010476 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:21.010449    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf6ac89_eb35_4d19_a28c_6f879f02703a.slice/crio-85aded1675877ff70bc4dcb6aadcfa8a55557c0e994618f8edbe4d3a564730f1 WatchSource:0}: Error finding container 85aded1675877ff70bc4dcb6aadcfa8a55557c0e994618f8edbe4d3a564730f1: Status 404 returned error can't find the container with id 85aded1675877ff70bc4dcb6aadcfa8a55557c0e994618f8edbe4d3a564730f1
Oct 30 03:56:21.309887 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.309854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:21.309887 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.309901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:21.310157 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:21.310008    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:56:21.310157 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:21.310008    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:56:21.310157 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:21.310113    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs podName:623e4b93-06e4-44f9-bd85-d9b663df4834 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:22.310069067 +0000 UTC m=+4.139293412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs") pod "network-metrics-daemon-b67qk" (UID: "623e4b93-06e4-44f9-bd85-d9b663df4834") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:56:21.310157 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:21.310156    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret podName:bc9a426a-587d-4611-9298-90fc4b4f1dc3 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:22.310138202 +0000 UTC m=+4.139362530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret") pod "global-pull-secret-syncer-d67gd" (UID: "bc9a426a-587d-4611-9298-90fc4b4f1dc3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:56:21.410761 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.410730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8xt\" (UniqueName: \"kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt\") pod \"network-check-target-k2wgk\" (UID: \"4c692538-d934-46cf-8c4c-d6a888dd00a1\") " pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:21.410912 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:21.410894    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:56:21.410969 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:21.410918    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:56:21.410969 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:21.410931    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cr8xt for pod openshift-network-diagnostics/network-check-target-k2wgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:56:21.411050 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:21.410989    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt podName:4c692538-d934-46cf-8c4c-d6a888dd00a1 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:22.410970945 +0000 UTC m=+4.240195289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cr8xt" (UniqueName: "kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt") pod "network-check-target-k2wgk" (UID: "4c692538-d934-46cf-8c4c-d6a888dd00a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:56:21.644294 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.644183    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 03:51:20 +0000 UTC" deadline="2028-06-11 10:25:55.608467131 +0000 UTC"
Oct 30 03:56:21.644294 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.644246    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14166h29m33.96422505s"
Oct 30 03:56:21.672622 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.671104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:21.672622 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:21.671224    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:21.683644 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.683545    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn" event={"ID":"478b6cf0-d0dd-4d65-aafb-ff211b60c188","Type":"ContainerStarted","Data":"2353c7b074a790f5e9c847cbc30bb3566ed8cdb8aafa871553776ea0972f8f75"}
Oct 30 03:56:21.688863 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.687987    2577 generic.go:356] "Generic (PLEG): container finished" podID="be906c4a667f460e0e8153479b6ef733" containerID="9ed1e03d302f1c5c4c59904fa3a9c88a3e8e60963b7a6273b82ec5a69c0365b1" exitCode=0
Oct 30 03:56:21.688863 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.688065    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal" event={"ID":"be906c4a667f460e0e8153479b6ef733","Type":"ContainerDied","Data":"9ed1e03d302f1c5c4c59904fa3a9c88a3e8e60963b7a6273b82ec5a69c0365b1"}
Oct 30 03:56:21.702450 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.702411    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf" event={"ID":"cbf6ac89-eb35-4d19-a28c-6f879f02703a","Type":"ContainerStarted","Data":"85aded1675877ff70bc4dcb6aadcfa8a55557c0e994618f8edbe4d3a564730f1"}
Oct 30 03:56:21.705464 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.705387    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qz4zz" event={"ID":"99e3f70c-6dd8-467e-a444-7692e30a76a4","Type":"ContainerStarted","Data":"98e094a82fa642764e6c2381a81397107484f1aa480bd5583094431442a6e5b2"}
Oct 30 03:56:21.729050 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.729016    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6czxj" event={"ID":"41551065-0510-4289-a742-83987e916774","Type":"ContainerStarted","Data":"e3f807feb80a7f1d40ed6617d546ee2855991902856e0a2a30ee79d0325386c7"}
Oct 30 03:56:21.730722 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.730639    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkl8" event={"ID":"9e6da480-8072-42d2-98f3-91566744aa96","Type":"ContainerStarted","Data":"89472633618f57002f7eef82f103ad7c8616a1475fe681f81d5a48004b399602"}
Oct 30 03:56:21.732236 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.732161    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mc77t" event={"ID":"0384e218-1937-4099-8566-67775cda0521","Type":"ContainerStarted","Data":"3f54ad826dc14af64a98e5eef31e80bd30ee7dd25712a14a85bad38fad43b133"}
Oct 30 03:56:21.736071 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.735479    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal" event={"ID":"f95db1d51f36d0a32fb67f82ccd47c76","Type":"ContainerStarted","Data":"01f2ebbcd6a8f15c546c36f0f801ebc4b4a86b09460cc4a9d4a9c9e65d59f048"}
Oct 30 03:56:21.737561 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.737531    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ttzr9" event={"ID":"c6fec406-0515-44b5-ae95-c2fbfe84b452","Type":"ContainerStarted","Data":"18eed020933973ad437dd0e5595f20105d27486ffa38d4da81dabf0cec261f3b"}
Oct 30 03:56:21.742270 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.742244    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d7nsr" event={"ID":"8e7b0fed-186b-4d7b-affa-e91d5ef4b523","Type":"ContainerStarted","Data":"de458eaf8dd33fa9b7e09166d8cfdf13201dbdece842f3396a4b802ec4da6d8b"}
Oct 30 03:56:21.745277 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:21.745237    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6njd4" event={"ID":"b555cf82-5423-4c47-a004-ae399654683b","Type":"ContainerStarted","Data":"ab712e529ca78f102a7a2ab9ce589176d2173167965d7792ab6dcf8e012e4f65"}
Oct 30 03:56:22.318821 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:22.318745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:22.318821 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:22.318806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:22.319035 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:22.318928    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:56:22.319035 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:22.318985    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret podName:bc9a426a-587d-4611-9298-90fc4b4f1dc3 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:24.318967741 +0000 UTC m=+6.148192086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret") pod "global-pull-secret-syncer-d67gd" (UID: "bc9a426a-587d-4611-9298-90fc4b4f1dc3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:56:22.319437 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:22.319397    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:56:22.319503 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:22.319453    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs podName:623e4b93-06e4-44f9-bd85-d9b663df4834 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:24.319436964 +0000 UTC m=+6.148661295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs") pod "network-metrics-daemon-b67qk" (UID: "623e4b93-06e4-44f9-bd85-d9b663df4834") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:56:22.419546 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:22.419515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8xt\" (UniqueName: \"kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt\") pod \"network-check-target-k2wgk\" (UID: \"4c692538-d934-46cf-8c4c-d6a888dd00a1\") " pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:22.419737 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:22.419718    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:56:22.419802 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:22.419747    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:56:22.419802 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:22.419759    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cr8xt for pod openshift-network-diagnostics/network-check-target-k2wgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:56:22.419887 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:22.419815    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt podName:4c692538-d934-46cf-8c4c-d6a888dd00a1 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:24.419794425 +0000 UTC m=+6.249018755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cr8xt" (UniqueName: "kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt") pod "network-check-target-k2wgk" (UID: "4c692538-d934-46cf-8c4c-d6a888dd00a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:56:22.644851 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:22.644784    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 03:51:20 +0000 UTC" deadline="2028-06-25 21:59:03.122339983 +0000 UTC"
Oct 30 03:56:22.644851 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:22.644822    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14514h2m40.477522945s"
Oct 30 03:56:22.673646 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:22.673618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:22.673795 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:22.673753    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:22.674185 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:22.674159    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:22.674300 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:22.674257    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:22.754751 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:22.754716    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal" event={"ID":"be906c4a667f460e0e8153479b6ef733","Type":"ContainerStarted","Data":"225e3cd06f922c77363973c49c4d5e18ad69d3eb36c455f997b6bf307969812d"}
Oct 30 03:56:22.796773 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:22.796716    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal" podStartSLOduration=2.796697945 podStartE2EDuration="2.796697945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:22.796519813 +0000 UTC m=+4.625744210" watchObservedRunningTime="2026-10-30 03:56:22.796697945 +0000 UTC m=+4.625922295"
Oct 30 03:56:22.797467 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:22.797424    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal" podStartSLOduration=2.797411548 podStartE2EDuration="2.797411548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:21.777714051 +0000 UTC m=+3.606938401" watchObservedRunningTime="2026-10-30 03:56:22.797411548 +0000 UTC m=+4.626635911"
Oct 30 03:56:23.680145 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:23.670758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:23.680145 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:23.670887    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:23.770662 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:23.770626    2577 generic.go:356] "Generic (PLEG): container finished" podID="9e6da480-8072-42d2-98f3-91566744aa96" containerID="2d30ee5d6ee7dec553ea9de36e3e7b99f058393972d84c3b07d1da7981303f09" exitCode=0
Oct 30 03:56:23.771684 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:23.771656    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkl8" event={"ID":"9e6da480-8072-42d2-98f3-91566744aa96","Type":"ContainerDied","Data":"2d30ee5d6ee7dec553ea9de36e3e7b99f058393972d84c3b07d1da7981303f09"}
Oct 30 03:56:24.336526 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:24.335892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:24.336526 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:24.335941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:24.336526 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:24.336097    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:56:24.336526 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:24.336099    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:56:24.336526 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:24.336153    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs podName:623e4b93-06e4-44f9-bd85-d9b663df4834 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:28.336137013 +0000 UTC m=+10.165361343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs") pod "network-metrics-daemon-b67qk" (UID: "623e4b93-06e4-44f9-bd85-d9b663df4834") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:56:24.336526 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:24.336172    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret podName:bc9a426a-587d-4611-9298-90fc4b4f1dc3 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:28.336162773 +0000 UTC m=+10.165387103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret") pod "global-pull-secret-syncer-d67gd" (UID: "bc9a426a-587d-4611-9298-90fc4b4f1dc3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:56:24.437239 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:24.436574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8xt\" (UniqueName: \"kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt\") pod \"network-check-target-k2wgk\" (UID: \"4c692538-d934-46cf-8c4c-d6a888dd00a1\") " pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:24.437239 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:24.436765    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:56:24.437239 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:24.436787    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:56:24.437239 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:24.436799    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cr8xt for pod openshift-network-diagnostics/network-check-target-k2wgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:56:24.437239 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:24.436849    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt podName:4c692538-d934-46cf-8c4c-d6a888dd00a1 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:28.436833556 +0000 UTC m=+10.266057889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cr8xt" (UniqueName: "kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt") pod "network-check-target-k2wgk" (UID: "4c692538-d934-46cf-8c4c-d6a888dd00a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:56:24.673386 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:24.673351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:24.673513 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:24.673486    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:24.673588 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:24.673549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:24.673639 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:24.673615    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:25.670832 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:25.670799    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:25.671420 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:25.670926    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:25.782958 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:25.782920    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d7nsr" event={"ID":"8e7b0fed-186b-4d7b-affa-e91d5ef4b523","Type":"ContainerStarted","Data":"bbdcf056f2b8274c5c616f313ffcfd53146b35e10c032eb94dd5ce3db315deea"}
Oct 30 03:56:25.826419 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:25.826349    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d7nsr" podStartSLOduration=4.117610507 podStartE2EDuration="7.826329183s" totalImagesPullingTime="3.708718676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:25.825326041 +0000 UTC m=+7.654550392" watchObservedRunningTime="2026-10-30 03:56:25.826329183 +0000 UTC m=+7.655553533"
Oct 30 03:56:26.671498 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:26.671469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:26.671922 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:26.671614    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:26.674294 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:26.674272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:26.674413 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:26.674393    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:26.801880 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:26.801332    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mc77t" event={"ID":"0384e218-1937-4099-8566-67775cda0521","Type":"ContainerStarted","Data":"68939897ef88a5b177485cc3d3372475c7310a29e96d1c5c3763c5fe8dde6b7b"}
Oct 30 03:56:27.671401 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:27.671369    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:27.671562 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:27.671480    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:27.808690 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:27.808660    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn" event={"ID":"478b6cf0-d0dd-4d65-aafb-ff211b60c188","Type":"ContainerStarted","Data":"0297db17bc6d7e9be82f2f29e64da75dbb9ee0cbe9abe45f3dbe404a397428d9"}
Oct 30 03:56:28.213649 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.213610    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 03:56:28.252971 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.252944    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:56:28.254231 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.254213    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:56:28.297221 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.297159    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mc77t" podStartSLOduration=5.120928802 podStartE2EDuration="10.297140049s" totalImagesPullingTime="5.176211247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:26.844444792 +0000 UTC m=+8.673669142" watchObservedRunningTime="2026-10-30 03:56:28.297140049 +0000 UTC m=+10.126364400"
Oct 30 03:56:28.371608 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.370494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:28.371608 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.370573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:28.371608 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:28.370706    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:56:28.371608 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:28.370764    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret podName:bc9a426a-587d-4611-9298-90fc4b4f1dc3 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:36.370746961 +0000 UTC m=+18.199971298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret") pod "global-pull-secret-syncer-d67gd" (UID: "bc9a426a-587d-4611-9298-90fc4b4f1dc3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:56:28.371608 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:28.371493    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:56:28.371608 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:28.371536    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs podName:623e4b93-06e4-44f9-bd85-d9b663df4834 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:36.37152368 +0000 UTC m=+18.200748008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs") pod "network-metrics-daemon-b67qk" (UID: "623e4b93-06e4-44f9-bd85-d9b663df4834") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:56:28.470982 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.470895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8xt\" (UniqueName: \"kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt\") pod \"network-check-target-k2wgk\" (UID: \"4c692538-d934-46cf-8c4c-d6a888dd00a1\") " pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:28.471251 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:28.471108    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:56:28.471251 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:28.471129    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:56:28.471251 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:28.471143    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cr8xt for pod openshift-network-diagnostics/network-check-target-k2wgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:56:28.471251 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:28.471192    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt podName:4c692538-d934-46cf-8c4c-d6a888dd00a1 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:36.471175797 +0000 UTC m=+18.300400131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cr8xt" (UniqueName: "kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt") pod "network-check-target-k2wgk" (UID: "4c692538-d934-46cf-8c4c-d6a888dd00a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:56:28.668537 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.667641    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T03:56:28.213635427Z","UUID":"70657342-0c73-4465-85fb-b668d4c9b314","Handler":null,"Name":"","Endpoint":""}
Oct 30 03:56:28.673681 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.670044    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 03:56:28.673681 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.670091    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 03:56:28.673681 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.672375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:28.673681 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:28.672500    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:28.673681 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.673448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:28.673681 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:28.673539    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:28.818197 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.818144    2577 generic.go:356] "Generic (PLEG): container finished" podID="9e6da480-8072-42d2-98f3-91566744aa96" containerID="fadc70239664b552305d3c91e36bf1b9b181bf5be2e6048d7c23529e315eb36d" exitCode=0
Oct 30 03:56:28.818340 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.818231    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkl8" event={"ID":"9e6da480-8072-42d2-98f3-91566744aa96","Type":"ContainerDied","Data":"fadc70239664b552305d3c91e36bf1b9b181bf5be2e6048d7c23529e315eb36d"}
Oct 30 03:56:28.823256 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.822758    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ttzr9" event={"ID":"c6fec406-0515-44b5-ae95-c2fbfe84b452","Type":"ContainerStarted","Data":"48123918153cc812165eb6f276acc8400f48f3b51628f6b51ece4af227438ef4"}
Oct 30 03:56:28.827229 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.827202    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn" event={"ID":"478b6cf0-d0dd-4d65-aafb-ff211b60c188","Type":"ContainerStarted","Data":"19255f902f1b0a9341a8630cdfa67de90b8c3899c440e8991d63d25b68b21fd4"}
Oct 30 03:56:28.914408 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:28.914345    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ttzr9" podStartSLOduration=3.302840299 podStartE2EDuration="10.914325675s" totalImagesPullingTime="7.611485376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:28.91334225 +0000 UTC m=+10.742566613" watchObservedRunningTime="2026-10-30 03:56:28.914325675 +0000 UTC m=+10.743550026"
Oct 30 03:56:29.671176 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:29.671096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:29.671338 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:29.671232    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:29.832679 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:29.832583    2577 generic.go:356] "Generic (PLEG): container finished" podID="9e6da480-8072-42d2-98f3-91566744aa96" containerID="ed1b70edab7f9941b8e230d418a8100d3a89afc88de05dd9e3e8a845046c7a67" exitCode=0
Oct 30 03:56:29.832679 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:29.832631    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkl8" event={"ID":"9e6da480-8072-42d2-98f3-91566744aa96","Type":"ContainerDied","Data":"ed1b70edab7f9941b8e230d418a8100d3a89afc88de05dd9e3e8a845046c7a67"}
Oct 30 03:56:29.835055 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:29.835026    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6njd4" event={"ID":"b555cf82-5423-4c47-a004-ae399654683b","Type":"ContainerStarted","Data":"2d875c03ea2e58b32f9be04d01f6b7778a5773541a6eda93057920d8e88fa068"}
Oct 30 03:56:29.840154 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:29.840125    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn" event={"ID":"478b6cf0-d0dd-4d65-aafb-ff211b60c188","Type":"ContainerStarted","Data":"e793d5f69d65a659f00fc949c0580c5067cdd282b7c7e9a4254d3acefa4944bf"}
Oct 30 03:56:29.933964 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:29.933863    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn" podStartSLOduration=3.9473848350000003 podStartE2EDuration="11.933845459s" totalImagesPullingTime="7.986460624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:29.933632525 +0000 UTC m=+11.762856875" watchObservedRunningTime="2026-10-30 03:56:29.933845459 +0000 UTC m=+11.763069810"
Oct 30 03:56:29.975629 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:29.975585    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6njd4" podStartSLOduration=4.200769271 podStartE2EDuration="11.975571662s" totalImagesPullingTime="7.774802391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:29.97511964 +0000 UTC m=+11.804343989" watchObservedRunningTime="2026-10-30 03:56:29.975571662 +0000 UTC m=+11.804796009"
Oct 30 03:56:30.671690 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:30.671656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:30.671845 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:30.671709    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:30.671917 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:30.671853    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:30.672018 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:30.671985    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:30.844050 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:30.844015    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6czxj" event={"ID":"41551065-0510-4289-a742-83987e916774","Type":"ContainerStarted","Data":"671a1442cbac1aff6fe1c77942225b7b81f737cfe2e83d668e865969225b6fe8"}
Oct 30 03:56:30.846828 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:30.846788    2577 generic.go:356] "Generic (PLEG): container finished" podID="9e6da480-8072-42d2-98f3-91566744aa96" containerID="2b616797926ac77fd542be14e0379d36d053bfcb4d2c572d834faea1fd9e33ae" exitCode=0
Oct 30 03:56:30.846970 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:30.846945    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkl8" event={"ID":"9e6da480-8072-42d2-98f3-91566744aa96","Type":"ContainerDied","Data":"2b616797926ac77fd542be14e0379d36d053bfcb4d2c572d834faea1fd9e33ae"}
Oct 30 03:56:31.671258 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:31.671174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:31.671409 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:31.671310    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:31.850381 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:31.850347    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qz4zz" event={"ID":"99e3f70c-6dd8-467e-a444-7692e30a76a4","Type":"ContainerStarted","Data":"4989746e46934b73e19177b9f08ea11f1317b9011bb0b0f2b1a9105166363463"}
Oct 30 03:56:31.897288 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:31.896419    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qz4zz" podStartSLOduration=2.6413061410000003 podStartE2EDuration="12.896399595s" totalImagesPullingTime="10.255093454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:31.895525678 +0000 UTC m=+13.724750029" watchObservedRunningTime="2026-10-30 03:56:31.896399595 +0000 UTC m=+13.725623942"
Oct 30 03:56:31.938103 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:31.938000    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6czxj" podStartSLOduration=4.846599811 podStartE2EDuration="12.937987866s" totalImagesPullingTime="8.091388055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:31.937440809 +0000 UTC m=+13.766665157" watchObservedRunningTime="2026-10-30 03:56:31.937987866 +0000 UTC m=+13.767212208"
Oct 30 03:56:32.673107 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:32.671438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:32.673107 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:32.671451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:32.673107 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:32.671576    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:32.673107 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:32.671660    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:33.671811 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:33.671596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:33.672323 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:33.671902    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:33.814421 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:33.814377    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:56:33.814575 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:33.814526    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:56:33.815338 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:33.815315    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:56:33.857564 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:33.857536    2577 generic.go:356] "Generic (PLEG): container finished" podID="9e6da480-8072-42d2-98f3-91566744aa96" containerID="f9a57bcc89b8f883b874d2c45e18f1a0cc3361e85f3e3794e155d0d82ab7b8d7" exitCode=0
Oct 30 03:56:33.858118 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:33.858099    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkl8" event={"ID":"9e6da480-8072-42d2-98f3-91566744aa96","Type":"ContainerDied","Data":"f9a57bcc89b8f883b874d2c45e18f1a0cc3361e85f3e3794e155d0d82ab7b8d7"}
Oct 30 03:56:34.671258 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:34.671231    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:34.671433 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:34.671240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:34.671433 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:34.671368    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:34.671433 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:34.671401    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:34.863058 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:34.863021    2577 generic.go:356] "Generic (PLEG): container finished" podID="9e6da480-8072-42d2-98f3-91566744aa96" containerID="7c80aef88f8f0c52c71371234c7c7f5c7f40ce1ed547052d09aa3ae0a01ba640" exitCode=0
Oct 30 03:56:34.863437 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:34.863098    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkl8" event={"ID":"9e6da480-8072-42d2-98f3-91566744aa96","Type":"ContainerDied","Data":"7c80aef88f8f0c52c71371234c7c7f5c7f40ce1ed547052d09aa3ae0a01ba640"}
Oct 30 03:56:35.671233 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:35.671032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:35.671428 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:35.671327    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:35.868680 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:35.868594    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf" event={"ID":"cbf6ac89-eb35-4d19-a28c-6f879f02703a","Type":"ContainerStarted","Data":"c1655748aa55ca5195e9fd5d084e74de5f9c6df27694706788dcbc02507c34d8"}
Oct 30 03:56:35.868680 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:35.868667    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf" event={"ID":"cbf6ac89-eb35-4d19-a28c-6f879f02703a","Type":"ContainerStarted","Data":"b89099dd2861c88c129f4160b2ae89cc6b253b6503d61311e4e8ccae191c4166"}
Oct 30 03:56:35.869165 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:35.868706    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf" event={"ID":"cbf6ac89-eb35-4d19-a28c-6f879f02703a","Type":"ContainerStarted","Data":"47be06dd619a698f8bd3561b9e8cc54fe10b6131a12fcad05a92df48137ddecd"}
Oct 30 03:56:35.869165 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:35.868734    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf" event={"ID":"cbf6ac89-eb35-4d19-a28c-6f879f02703a","Type":"ContainerStarted","Data":"075c6c4ce08e1b8aee494cc944f21ed06ef522e9a7df97004515fc71765ff520"}
Oct 30 03:56:35.869165 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:35.868760    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf" event={"ID":"cbf6ac89-eb35-4d19-a28c-6f879f02703a","Type":"ContainerStarted","Data":"b0156ac97f0770baaf06454d87473ed48ae430611cd70bce021a9ab412e3b9f5"}
Oct 30 03:56:35.869165 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:35.868799    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf" event={"ID":"cbf6ac89-eb35-4d19-a28c-6f879f02703a","Type":"ContainerStarted","Data":"0b4d5bae94d9f9492ef30b35b6738c234307a0ad1b2268c32a910c843bb69882"}
Oct 30 03:56:35.871884 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:35.871848    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkl8" event={"ID":"9e6da480-8072-42d2-98f3-91566744aa96","Type":"ContainerStarted","Data":"213c25beb1e09fce9b33dfeb1d39171f5f1b25a7e30f77947a302e8522e1ee6b"}
Oct 30 03:56:35.921115 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:35.921054    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2gkl8" podStartSLOduration=6.79541163 podStartE2EDuration="16.921043355s" totalImagesPullingTime="10.125631725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:35.920879039 +0000 UTC m=+17.750103387" watchObservedRunningTime="2026-10-30 03:56:35.921043355 +0000 UTC m=+17.750267704"
Oct 30 03:56:36.432326 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:36.432299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:36.432566 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:36.432337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:36.432566 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:36.432449    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:56:36.432566 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:36.432457    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:56:36.432566 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:36.432499    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret podName:bc9a426a-587d-4611-9298-90fc4b4f1dc3 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:52.432485472 +0000 UTC m=+34.261709799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret") pod "global-pull-secret-syncer-d67gd" (UID: "bc9a426a-587d-4611-9298-90fc4b4f1dc3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:56:36.432566 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:36.432521    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs podName:623e4b93-06e4-44f9-bd85-d9b663df4834 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:52.432505669 +0000 UTC m=+34.261729998 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs") pod "network-metrics-daemon-b67qk" (UID: "623e4b93-06e4-44f9-bd85-d9b663df4834") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:56:36.533035 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:36.533007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8xt\" (UniqueName: \"kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt\") pod \"network-check-target-k2wgk\" (UID: \"4c692538-d934-46cf-8c4c-d6a888dd00a1\") " pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:36.533147 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:36.533141    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:56:36.533181 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:36.533153    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:56:36.533181 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:36.533162    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cr8xt for pod openshift-network-diagnostics/network-check-target-k2wgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:56:36.533252 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:36.533203    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt podName:4c692538-d934-46cf-8c4c-d6a888dd00a1 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:52.533188947 +0000 UTC m=+34.362413276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cr8xt" (UniqueName: "kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt") pod "network-check-target-k2wgk" (UID: "4c692538-d934-46cf-8c4c-d6a888dd00a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:56:36.671399 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:36.671364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:36.671399 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:36.671392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:36.671551 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:36.671479    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:36.671617 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:36.671598    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:37.670685 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:37.670663    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:37.671039 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:37.670753    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:37.879614 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:37.879579    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf" event={"ID":"cbf6ac89-eb35-4d19-a28c-6f879f02703a","Type":"ContainerStarted","Data":"a1e86b67f8e6de5166a05aa58c62005aa3c943a5cf4043a91b3f301929df8440"}
Oct 30 03:56:38.671381 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:38.671357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:38.671813 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:38.671500    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:38.671813 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:38.671535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:38.671813 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:38.671611    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:39.671296 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:39.671268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:39.671441 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:39.671354    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:39.886836 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:39.886798    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf" event={"ID":"cbf6ac89-eb35-4d19-a28c-6f879f02703a","Type":"ContainerStarted","Data":"15dfd016250383b29bff94847a1d688600cd7c7927ed2cc8bb145d9fd8ef9a7a"}
Oct 30 03:56:39.887133 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:39.887113    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:39.887437 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:39.887417    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:39.903678 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:39.903515    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:39.903787 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:39.903758    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:39.984744 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:39.984695    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf" podStartSLOduration=6.701487205 podStartE2EDuration="20.984683497s" totalImagesPullingTime="14.283196292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:39.984246294 +0000 UTC m=+21.813470645" watchObservedRunningTime="2026-10-30 03:56:39.984683497 +0000 UTC m=+21.813907846"
Oct 30 03:56:40.670993 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:40.670962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:40.671194 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:40.671114    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:40.671194 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:40.671173    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:40.671308 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:40.671285    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:40.888723 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:40.888701    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:56:41.671641 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:41.671615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:41.671844 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:41.671711    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:41.819438 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:41.819409    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d67gd"]
Oct 30 03:56:41.819567 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:41.819504    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:41.819614 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:41.819581    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:41.822341 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:41.822313    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k2wgk"]
Oct 30 03:56:41.833396 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:41.833374    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b67qk"]
Oct 30 03:56:41.833472 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:41.833456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:41.833543 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:41.833527    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:41.890575 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:41.890551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:41.890857 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:41.890655    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:56:41.890857 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:41.890674    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:43.218554 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:43.218517    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:56:43.671038 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:43.670968    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:43.671038 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:43.670995    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:43.671214 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:43.670971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:43.671214 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:43.671060    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:43.671214 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:43.671178    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:43.671305 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:43.671223    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:45.671327 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:45.671294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:45.671865 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:45.671294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:45.671865 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:45.671418    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:45.671865 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:45.671294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:45.671865 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:45.671527    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:45.671865 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:45.671465    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:47.671003 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:47.670962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:47.671003 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:47.671011    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:47.671525 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:47.671121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:47.671525 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:47.671129    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b67qk" podUID="623e4b93-06e4-44f9-bd85-d9b663df4834"
Oct 30 03:56:47.671525 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:47.671191    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d67gd" podUID="bc9a426a-587d-4611-9298-90fc4b4f1dc3"
Oct 30 03:56:47.671525 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:47.671259    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2wgk" podUID="4c692538-d934-46cf-8c4c-d6a888dd00a1"
Oct 30 03:56:48.452119 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.452069    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeReady"
Oct 30 03:56:48.452248 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.452186    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 03:56:48.503151 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.503122    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jmlk2"]
Oct 30 03:56:48.508405 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.508388    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-drvrv"]
Oct 30 03:56:48.508510 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.508494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jmlk2"
Oct 30 03:56:48.511741 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.511588    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7tc9g"]
Oct 30 03:56:48.511903 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.511879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-drvrv"
Oct 30 03:56:48.514022 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.514005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7tc9g"
Oct 30 03:56:48.519404 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.517066    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2"]
Oct 30 03:56:48.519757 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.519740    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xkq6m"]
Oct 30 03:56:48.519855 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.519839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2"
Oct 30 03:56:48.523277 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.523258    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"]
Oct 30 03:56:48.523387 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.523370    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:48.527319 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.526519    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-trv7j"]
Oct 30 03:56:48.527319 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.526668    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:56:48.532127 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.532108    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"]
Oct 30 03:56:48.532235 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.532164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.532435 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.532408    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 03:56:48.532862 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.532828    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 03:56:48.535963 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.535940    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"]
Oct 30 03:56:48.536055 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.536017    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:48.539566 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.539535    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-84bd8c5fd7-885hp"]
Oct 30 03:56:48.543957 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.543915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:56:48.546743 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.545251    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 03:56:48.546743 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.546634    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b2qtm\""
Oct 30 03:56:48.548135 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.548115    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"]
Oct 30 03:56:48.548237 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.548137    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.550188 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.550168    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 03:56:48.550286 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.550206    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 03:56:48.553906 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.553891    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-664tl"]
Oct 30 03:56:48.554060 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.554044    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:48.556107 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.556023    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 03:56:48.562545 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.557475    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-htskn"]
Oct 30 03:56:48.562545 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.558304    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.566357 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.565711    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bbcfd588b-rlkpf"]
Oct 30 03:56:48.566357 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.565819    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:56:48.569743 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.569725    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-d29md"]
Oct 30 03:56:48.569942 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.569924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.572682 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.572658    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-759fbb47f9-s2s2f"]
Oct 30 03:56:48.573180 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.573161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-d29md"
Oct 30 03:56:48.575848 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.575828    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bh464"]
Oct 30 03:56:48.575961 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.575946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.579425    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7tc9g"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.579464    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jmlk2"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.579486    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-drvrv"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.579505    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.579524    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xkq6m"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.579527    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bh464"
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.579550    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.579576    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-trv7j"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.579593    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-664tl"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.580179    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.580210    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.580231    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84bd8c5fd7-885hp"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.580254    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bbcfd588b-rlkpf"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.580276    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.580295    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bh464"]
Oct 30 03:56:48.582121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.580362    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-htskn"]
Oct 30 03:56:48.591749 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.591729    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-d29md"]
Oct 30 03:56:48.594724 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.594704    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-759fbb47f9-s2s2f"]
Oct 30 03:56:48.621947 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.621922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cktnq\" (UniqueName: \"kubernetes.io/projected/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-kube-api-access-cktnq\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:56:48.622048 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.621961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9z28\" (UniqueName: \"kubernetes.io/projected/788f5892-601d-42cc-b597-528a091feac9-kube-api-access-n9z28\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:48.622048 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.621995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-config-volume\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:48.622277 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:48.622277 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-tmp-dir\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:48.622277 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q52f\" (UniqueName: \"kubernetes.io/projected/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-kube-api-access-7q52f\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:48.622277 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d41c0686-14d0-43c4-928e-e38b57021fb7-snapshots\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.622460 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:48.622460 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1fe5ff28-0896-4f54-b5cc-5326a0d60752-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:56:48.622460 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twx6b\" (UniqueName: \"kubernetes.io/projected/1fe5ff28-0896-4f54-b5cc-5326a0d60752-kube-api-access-twx6b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:56:48.622460 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-default-certificate\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.622460 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppk4b\" (UniqueName: \"kubernetes.io/projected/d18cc860-7cd8-48d9-9a7a-40a40a53f379-kube-api-access-ppk4b\") pod \"kas-connection-checker-685895c7c9-7tc9g\" (UID: \"d18cc860-7cd8-48d9-9a7a-40a40a53f379\") " pod="kube-system/kas-connection-checker-685895c7c9-7tc9g"
Oct 30 03:56:48.622627 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28tfx\" (UniqueName: \"kubernetes.io/projected/4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b-kube-api-access-28tfx\") pod \"volume-data-source-validator-58c9f5f46d-cp4k2\" (UID: \"4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2"
Oct 30 03:56:48.622627 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.622627 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-stats-auth\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.622627 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d41c0686-14d0-43c4-928e-e38b57021fb7-tmp\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.622759 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp4k9\" (UniqueName: \"kubernetes.io/projected/d41c0686-14d0-43c4-928e-e38b57021fb7-kube-api-access-bp4k9\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.622759 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n278c\" (UniqueName: \"kubernetes.io/projected/b752c121-5e44-4df2-9f17-97f0c899516b-kube-api-access-n278c\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.622865 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fnjr\" (UniqueName: \"kubernetes.io/projected/5e68d10c-d217-40d3-945d-c87ccbd7434b-kube-api-access-9fnjr\") pod \"kas-connection-checker-685895c7c9-drvrv\" (UID: \"5e68d10c-d217-40d3-945d-c87ccbd7434b\") " pod="kube-system/kas-connection-checker-685895c7c9-drvrv"
Oct 30 03:56:48.622904 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-serving-cert\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:56:48.622941 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d41c0686-14d0-43c4-928e-e38b57021fb7-service-ca-bundle\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.622971 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.622958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d41c0686-14d0-43c4-928e-e38b57021fb7-serving-cert\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.623015 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.623002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxmnr\" (UniqueName: \"kubernetes.io/projected/4455e2df-ec71-4748-8920-36b2374af2a1-kube-api-access-gxmnr\") pod \"kas-connection-checker-685895c7c9-jmlk2\" (UID: \"4455e2df-ec71-4748-8920-36b2374af2a1\") " pod="kube-system/kas-connection-checker-685895c7c9-jmlk2"
Oct 30 03:56:48.623046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.623032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-config\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:56:48.623096 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.623056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d41c0686-14d0-43c4-928e-e38b57021fb7-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.623138 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.623105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/788f5892-601d-42cc-b597-528a091feac9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:48.623169 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.623132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1fe5ff28-0896-4f54-b5cc-5326a0d60752-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:56:48.623169 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.623157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.632007 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.631971    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 03:56:48.632121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632015    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 03:56:48.632121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632024    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:56:48.632121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632026    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:56:48.632121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632101    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:56:48.632121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632118    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 03:56:48.632422 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632176    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 03:56:48.632422 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632185    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:56:48.632422 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632266    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 03:56:48.632422 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632291    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 03:56:48.632592 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632447    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 03:56:48.632592 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632450    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 03:56:48.632592 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632466    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 03:56:48.632592 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632480    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 03:56:48.632592 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632482    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 03:56:48.632592 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632520    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:56:48.632592 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632529    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 03:56:48.632592 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632541    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:56:48.632592 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632546    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:56:48.632939 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.632644    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 03:56:48.633152 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.633128    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 03:56:48.633152 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.633148    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 03:56:48.633262 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.633180    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 03:56:48.633262 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.633242    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 03:56:48.633315 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.633270    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 03:56:48.633361 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.633347    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 03:56:48.633531 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.633519    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 03:56:48.633623 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.633562    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 03:56:48.646938 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.646917    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 03:56:48.660869 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.660482    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 03:56:48.660869 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.660648    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 03:56:48.660869 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.660719    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 03:56:48.660869 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.660758    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 03:56:48.660869 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.660820    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 03:56:48.660869 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.660824    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8wd8q\""
Oct 30 03:56:48.660869 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.660867    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 03:56:48.661415 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.660878    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 03:56:48.661415 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.660907    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 03:56:48.661415 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.660935    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 03:56:48.661415 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.660944    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 03:56:48.661415 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.661005    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 03:56:48.661415 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.661112    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 03:56:48.661415 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.661230    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 03:56:48.661415 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.661317    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hmk7j\""
Oct 30 03:56:48.661415 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.661320    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 03:56:48.723472 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.723401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28tfx\" (UniqueName: \"kubernetes.io/projected/4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b-kube-api-access-28tfx\") pod \"volume-data-source-validator-58c9f5f46d-cp4k2\" (UID: \"4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2"
Oct 30 03:56:48.723472 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.723433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.723472 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.723461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/907912b8-9313-43b3-b324-e8498ea6519a-config\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.723923 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.723617    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle podName:b752c121-5e44-4df2-9f17-97f0c899516b nodeName:}" failed. No retries permitted until 2026-10-30 03:56:49.223596142 +0000 UTC m=+31.052820484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle") pod "router-default-84bd8c5fd7-885hp" (UID: "b752c121-5e44-4df2-9f17-97f0c899516b") : configmap references non-existent config key: service-ca.crt
Oct 30 03:56:48.723923 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.723660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d41c0686-14d0-43c4-928e-e38b57021fb7-tmp\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.723923 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.723699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99bzl\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-kube-api-access-99bzl\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.723923 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.723726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/343419d3-210d-4c12-86ff-c92f8f574b27-ca-trust-extracted\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.723923 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.723771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d41c0686-14d0-43c4-928e-e38b57021fb7-serving-cert\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.723923 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.723817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-image-registry-private-configuration\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.724232 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.723942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fnjr\" (UniqueName: \"kubernetes.io/projected/5e68d10c-d217-40d3-945d-c87ccbd7434b-kube-api-access-9fnjr\") pod \"kas-connection-checker-685895c7c9-drvrv\" (UID: \"5e68d10c-d217-40d3-945d-c87ccbd7434b\") " pod="kube-system/kas-connection-checker-685895c7c9-drvrv"
Oct 30 03:56:48.724232 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.723981    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d41c0686-14d0-43c4-928e-e38b57021fb7-tmp\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.724232 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d41c0686-14d0-43c4-928e-e38b57021fb7-service-ca-bundle\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.724232 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-bound-sa-token\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.724232 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-registry-certificates\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.724232 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-config\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:56:48.724232 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89f2993a-ed5d-488e-832d-bcba8ddd9706-cert\") pod \"ingress-canary-bh464\" (UID: \"89f2993a-ed5d-488e-832d-bcba8ddd9706\") " pod="openshift-ingress-canary/ingress-canary-bh464"
Oct 30 03:56:48.724445 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/907912b8-9313-43b3-b324-e8498ea6519a-trusted-ca\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.724445 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-config-volume\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:48.724445 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-tmp-dir\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:48.724445 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:48.724618 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twx6b\" (UniqueName: \"kubernetes.io/projected/1fe5ff28-0896-4f54-b5cc-5326a0d60752-kube-api-access-twx6b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:56:48.724618 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-image-registry-private-configuration\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.724618 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppk4b\" (UniqueName: \"kubernetes.io/projected/d18cc860-7cd8-48d9-9a7a-40a40a53f379-kube-api-access-ppk4b\") pod \"kas-connection-checker-685895c7c9-7tc9g\" (UID: \"d18cc860-7cd8-48d9-9a7a-40a40a53f379\") " pod="kube-system/kas-connection-checker-685895c7c9-7tc9g"
Oct 30 03:56:48.724618 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.724618 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-bound-sa-token\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.724618 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.724618 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-stats-auth\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.724942 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp4k9\" (UniqueName: \"kubernetes.io/projected/d41c0686-14d0-43c4-928e-e38b57021fb7-kube-api-access-bp4k9\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.724942 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724643    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d41c0686-14d0-43c4-928e-e38b57021fb7-service-ca-bundle\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.724942 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n278c\" (UniqueName: \"kubernetes.io/projected/b752c121-5e44-4df2-9f17-97f0c899516b-kube-api-access-n278c\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.724942 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724664    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-config\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:56:48.724942 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-trusted-ca\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.724942 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724701    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-tmp-dir\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:48.724942 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9tt\" (UniqueName: \"kubernetes.io/projected/89f2993a-ed5d-488e-832d-bcba8ddd9706-kube-api-access-9b9tt\") pod \"ingress-canary-bh464\" (UID: \"89f2993a-ed5d-488e-832d-bcba8ddd9706\") " pod="openshift-ingress-canary/ingress-canary-bh464"
Oct 30 03:56:48.724942 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.724782    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:56:48.724942 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.724844    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs podName:b752c121-5e44-4df2-9f17-97f0c899516b nodeName:}" failed. No retries permitted until 2026-10-30 03:56:49.224827967 +0000 UTC m=+31.054052298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs") pod "router-default-84bd8c5fd7-885hp" (UID: "b752c121-5e44-4df2-9f17-97f0c899516b") : secret "router-metrics-certs-default" not found
Oct 30 03:56:48.724942 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724891    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-config-volume\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/788f5892-601d-42cc-b597-528a091feac9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.724981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-trusted-ca\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdlq\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-kube-api-access-vzdlq\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.725020    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tnhw\" (UniqueName: \"kubernetes.io/projected/1d40d8b3-4844-4a76-b395-b5b8a2a41438-kube-api-access-6tnhw\") pod \"network-check-source-6f78d59f99-d29md\" (UID: \"1d40d8b3-4844-4a76-b395-b5b8a2a41438\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-d29md"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-serving-cert\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.725100    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls podName:788f5892-601d-42cc-b597-528a091feac9 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:49.225064799 +0000 UTC m=+31.054289148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xccsz" (UID: "788f5892-601d-42cc-b597-528a091feac9") : secret "samples-operator-tls" not found
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/503b82e1-e56c-4c80-9d63-fa8e647c0991-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mq42\" (UniqueName: \"kubernetes.io/projected/907912b8-9313-43b3-b324-e8498ea6519a-kube-api-access-5mq42\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/907912b8-9313-43b3-b324-e8498ea6519a-serving-cert\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d41c0686-14d0-43c4-928e-e38b57021fb7-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1fe5ff28-0896-4f54-b5cc-5326a0d60752-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-installation-pull-secrets\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.725493 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/395d5cbd-c077-4578-b377-3034e42a2b94-ca-trust-extracted\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-installation-pull-secrets\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktnq\" (UniqueName: \"kubernetes.io/projected/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-kube-api-access-cktnq\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-registry-certificates\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725618    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/788f5892-601d-42cc-b597-528a091feac9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9z28\" (UniqueName: \"kubernetes.io/projected/788f5892-601d-42cc-b597-528a091feac9-kube-api-access-n9z28\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmnr\" (UniqueName: \"kubernetes.io/projected/4455e2df-ec71-4748-8920-36b2374af2a1-kube-api-access-gxmnr\") pod \"kas-connection-checker-685895c7c9-jmlk2\" (UID: \"4455e2df-ec71-4748-8920-36b2374af2a1\") " pod="kube-system/kas-connection-checker-685895c7c9-jmlk2"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.725913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.726057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1fe5ff28-0896-4f54-b5cc-5326a0d60752-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.726151    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.726183    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls podName:730fcc37-b18e-46e3-aea5-a7fba60b7a8c nodeName:}" failed. No retries permitted until 2026-10-30 03:56:49.22617147 +0000 UTC m=+31.055395797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls") pod "dns-default-xkq6m" (UID: "730fcc37-b18e-46e3-aea5-a7fba60b7a8c") : secret "dns-default-metrics-tls" not found
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.726226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-default-certificate\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.726256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q52f\" (UniqueName: \"kubernetes.io/projected/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-kube-api-access-7q52f\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.726280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d41c0686-14d0-43c4-928e-e38b57021fb7-snapshots\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.726388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.726307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27j9s\" (UniqueName: \"kubernetes.io/projected/503b82e1-e56c-4c80-9d63-fa8e647c0991-kube-api-access-27j9s\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:48.727068 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.726338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:56:48.727068 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.726406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-nginx-conf\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:56:48.727068 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.726421    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d41c0686-14d0-43c4-928e-e38b57021fb7-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.727068 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.726671    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1fe5ff28-0896-4f54-b5cc-5326a0d60752-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:56:48.727068 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.726842    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d41c0686-14d0-43c4-928e-e38b57021fb7-snapshots\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.728065 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.728034    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-serving-cert\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:56:48.728200 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.728131    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1fe5ff28-0896-4f54-b5cc-5326a0d60752-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:56:48.728265 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.728244    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d41c0686-14d0-43c4-928e-e38b57021fb7-serving-cert\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.728323 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.728309    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-stats-auth\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.728817 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.728801    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-default-certificate\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.751130 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.751109    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tfx\" (UniqueName: \"kubernetes.io/projected/4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b-kube-api-access-28tfx\") pod \"volume-data-source-validator-58c9f5f46d-cp4k2\" (UID: \"4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2"
Oct 30 03:56:48.755372 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.755342    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fnjr\" (UniqueName: \"kubernetes.io/projected/5e68d10c-d217-40d3-945d-c87ccbd7434b-kube-api-access-9fnjr\") pod \"kas-connection-checker-685895c7c9-drvrv\" (UID: \"5e68d10c-d217-40d3-945d-c87ccbd7434b\") " pod="kube-system/kas-connection-checker-685895c7c9-drvrv"
Oct 30 03:56:48.755554 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.755526    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twx6b\" (UniqueName: \"kubernetes.io/projected/1fe5ff28-0896-4f54-b5cc-5326a0d60752-kube-api-access-twx6b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:56:48.756057 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.756038    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp4k9\" (UniqueName: \"kubernetes.io/projected/d41c0686-14d0-43c4-928e-e38b57021fb7-kube-api-access-bp4k9\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.756179 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.756163    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppk4b\" (UniqueName: \"kubernetes.io/projected/d18cc860-7cd8-48d9-9a7a-40a40a53f379-kube-api-access-ppk4b\") pod \"kas-connection-checker-685895c7c9-7tc9g\" (UID: \"d18cc860-7cd8-48d9-9a7a-40a40a53f379\") " pod="kube-system/kas-connection-checker-685895c7c9-7tc9g"
Oct 30 03:56:48.756716 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.756695    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmnr\" (UniqueName: \"kubernetes.io/projected/4455e2df-ec71-4748-8920-36b2374af2a1-kube-api-access-gxmnr\") pod \"kas-connection-checker-685895c7c9-jmlk2\" (UID: \"4455e2df-ec71-4748-8920-36b2374af2a1\") " pod="kube-system/kas-connection-checker-685895c7c9-jmlk2"
Oct 30 03:56:48.756775 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.756713    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n278c\" (UniqueName: \"kubernetes.io/projected/b752c121-5e44-4df2-9f17-97f0c899516b-kube-api-access-n278c\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:48.757933 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.757913    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q52f\" (UniqueName: \"kubernetes.io/projected/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-kube-api-access-7q52f\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:48.758002 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.757976    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cktnq\" (UniqueName: \"kubernetes.io/projected/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-kube-api-access-cktnq\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:56:48.758037 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.758019    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9z28\" (UniqueName: \"kubernetes.io/projected/788f5892-601d-42cc-b597-528a091feac9-kube-api-access-n9z28\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:48.825791 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.825770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jmlk2"
Oct 30 03:56:48.827068 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-image-registry-private-configuration\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.827177 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-bound-sa-token\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.827177 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-registry-certificates\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.827177 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89f2993a-ed5d-488e-832d-bcba8ddd9706-cert\") pod \"ingress-canary-bh464\" (UID: \"89f2993a-ed5d-488e-832d-bcba8ddd9706\") " pod="openshift-ingress-canary/ingress-canary-bh464"
Oct 30 03:56:48.827327 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/907912b8-9313-43b3-b324-e8498ea6519a-trusted-ca\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.827469 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-image-registry-private-configuration\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.827573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.827573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-bound-sa-token\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.827573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-trusted-ca\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.827573 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9tt\" (UniqueName: \"kubernetes.io/projected/89f2993a-ed5d-488e-832d-bcba8ddd9706-kube-api-access-9b9tt\") pod \"ingress-canary-bh464\" (UID: \"89f2993a-ed5d-488e-832d-bcba8ddd9706\") " pod="openshift-ingress-canary/ingress-canary-bh464"
Oct 30 03:56:48.827894 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-trusted-ca\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.827894 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdlq\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-kube-api-access-vzdlq\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.827894 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tnhw\" (UniqueName: \"kubernetes.io/projected/1d40d8b3-4844-4a76-b395-b5b8a2a41438-kube-api-access-6tnhw\") pod \"network-check-source-6f78d59f99-d29md\" (UID: \"1d40d8b3-4844-4a76-b395-b5b8a2a41438\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-d29md"
Oct 30 03:56:48.827894 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/503b82e1-e56c-4c80-9d63-fa8e647c0991-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:48.827894 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mq42\" (UniqueName: \"kubernetes.io/projected/907912b8-9313-43b3-b324-e8498ea6519a-kube-api-access-5mq42\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.827894 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/907912b8-9313-43b3-b324-e8498ea6519a-serving-cert\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.827894 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-installation-pull-secrets\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.827894 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827760    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-registry-certificates\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.827894 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.827894 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:48.827894 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/395d5cbd-c077-4578-b377-3034e42a2b94-ca-trust-extracted\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.827894 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-installation-pull-secrets\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.828427 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-registry-certificates\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.828427 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.827964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27j9s\" (UniqueName: \"kubernetes.io/projected/503b82e1-e56c-4c80-9d63-fa8e647c0991-kube-api-access-27j9s\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:48.828825 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.828718    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/503b82e1-e56c-4c80-9d63-fa8e647c0991-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:48.828982 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.828956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:56:48.829062 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.829000    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-trusted-ca\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.829062 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.829004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-nginx-conf\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:56:48.829218 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.829090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/907912b8-9313-43b3-b324-e8498ea6519a-config\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.829218 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.829120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99bzl\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-kube-api-access-99bzl\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.829218 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.829147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/343419d3-210d-4c12-86ff-c92f8f574b27-ca-trust-extracted\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.829750    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/343419d3-210d-4c12-86ff-c92f8f574b27-ca-trust-extracted\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.830159    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-nginx-conf\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.830377    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-image-registry-private-configuration\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.830539    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-trusted-ca\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.827592    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.830580    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-759fbb47f9-s2s2f: secret "image-registry-tls" not found
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.827876    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.830588    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-registry-certificates\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.830594    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/907912b8-9313-43b3-b324-e8498ea6519a-trusted-ca\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.830601    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbcfd588b-rlkpf: secret "image-registry-tls" not found
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.829453    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.830307    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.830639    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls podName:395d5cbd-c077-4578-b377-3034e42a2b94 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:49.330621615 +0000 UTC m=+31.159845961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls") pod "image-registry-759fbb47f9-s2s2f" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94") : secret "image-registry-tls" not found
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.830659    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls podName:503b82e1-e56c-4c80-9d63-fa8e647c0991 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:49.330649563 +0000 UTC m=+31.159873891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sb794" (UID: "503b82e1-e56c-4c80-9d63-fa8e647c0991") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.830659    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/907912b8-9313-43b3-b324-e8498ea6519a-serving-cert\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.830689    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls podName:343419d3-210d-4c12-86ff-c92f8f574b27 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:49.330680336 +0000 UTC m=+31.159904669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls") pod "image-registry-6bbcfd588b-rlkpf" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27") : secret "image-registry-tls" not found
Oct 30 03:56:48.830745 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.830576    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/395d5cbd-c077-4578-b377-3034e42a2b94-ca-trust-extracted\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.831765 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:48.830720    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert podName:01bc8bd9-f593-4acf-97c0-fc6c4beb4867 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:49.330711787 +0000 UTC m=+31.159936115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htskn" (UID: "01bc8bd9-f593-4acf-97c0-fc6c4beb4867") : secret "networking-console-plugin-cert" not found
Oct 30 03:56:48.831765 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.831193    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/907912b8-9313-43b3-b324-e8498ea6519a-config\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.831765 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.831707    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-image-registry-private-configuration\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.832094 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.832044    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-installation-pull-secrets\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.833210 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.833190    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-installation-pull-secrets\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.833511 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.833491    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89f2993a-ed5d-488e-832d-bcba8ddd9706-cert\") pod \"ingress-canary-bh464\" (UID: \"89f2993a-ed5d-488e-832d-bcba8ddd9706\") " pod="openshift-ingress-canary/ingress-canary-bh464"
Oct 30 03:56:48.833588 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.833574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-drvrv"
Oct 30 03:56:48.848355 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.848335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7tc9g"
Oct 30 03:56:48.852092 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.852060    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-bound-sa-token\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.852740 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.852721    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9tt\" (UniqueName: \"kubernetes.io/projected/89f2993a-ed5d-488e-832d-bcba8ddd9706-kube-api-access-9b9tt\") pod \"ingress-canary-bh464\" (UID: \"89f2993a-ed5d-488e-832d-bcba8ddd9706\") " pod="openshift-ingress-canary/ingress-canary-bh464"
Oct 30 03:56:48.852797 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.852745    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-bound-sa-token\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.852797 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.852721    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdlq\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-kube-api-access-vzdlq\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:48.852865 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.852845    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mq42\" (UniqueName: \"kubernetes.io/projected/907912b8-9313-43b3-b324-e8498ea6519a-kube-api-access-5mq42\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.855257 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.855235    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99bzl\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-kube-api-access-99bzl\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:48.855352 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.855336    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tnhw\" (UniqueName: \"kubernetes.io/projected/1d40d8b3-4844-4a76-b395-b5b8a2a41438-kube-api-access-6tnhw\") pod \"network-check-source-6f78d59f99-d29md\" (UID: \"1d40d8b3-4844-4a76-b395-b5b8a2a41438\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-d29md"
Oct 30 03:56:48.855465 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.855449    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27j9s\" (UniqueName: \"kubernetes.io/projected/503b82e1-e56c-4c80-9d63-fa8e647c0991-kube-api-access-27j9s\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:48.856147 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.856130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2"
Oct 30 03:56:48.874721 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.874705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:56:48.885043 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.885023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:56:48.920922 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.920879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:56:48.929855 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.929788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:48.958570 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.958505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-d29md"
Oct 30 03:56:48.969427 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:48.969400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bh464"
Oct 30 03:56:49.064205 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.064002    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:49.191826 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.191801    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-drvrv"]
Oct 30 03:56:49.193468 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:49.193430    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e68d10c_d217_40d3_945d_c87ccbd7434b.slice/crio-c3eb51613da26058dac2cbc5e835f79a7d4ac04cfae6ab2e2184d5017fde4878 WatchSource:0}: Error finding container c3eb51613da26058dac2cbc5e835f79a7d4ac04cfae6ab2e2184d5017fde4878: Status 404 returned error can't find the container with id c3eb51613da26058dac2cbc5e835f79a7d4ac04cfae6ab2e2184d5017fde4878
Oct 30 03:56:49.195015 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.194992    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jmlk2"]
Oct 30 03:56:49.195502 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:49.195434    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4455e2df_ec71_4748_8920_36b2374af2a1.slice/crio-a11b2554a66ce72684088750cca0389fa8e7e1900e259271cf46e2db0aeb7acc WatchSource:0}: Error finding container a11b2554a66ce72684088750cca0389fa8e7e1900e259271cf46e2db0aeb7acc: Status 404 returned error can't find the container with id a11b2554a66ce72684088750cca0389fa8e7e1900e259271cf46e2db0aeb7acc
Oct 30 03:56:49.211842 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.211808    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-trv7j"]
Oct 30 03:56:49.242030 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.242008    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2"]
Oct 30 03:56:49.244819 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.244572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:49.244819 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.244680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:49.244819 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.244699    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:56:49.244819 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:49.244699    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0cc1cb_a036_4ddb_92c1_8a900fc7bc4b.slice/crio-3f6923c74cb8d51b74345c2b62cf915662d2c876238e315d471e4308bfcc82fb WatchSource:0}: Error finding container 3f6923c74cb8d51b74345c2b62cf915662d2c876238e315d471e4308bfcc82fb: Status 404 returned error can't find the container with id 3f6923c74cb8d51b74345c2b62cf915662d2c876238e315d471e4308bfcc82fb
Oct 30 03:56:49.245479 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.245187    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls podName:730fcc37-b18e-46e3-aea5-a7fba60b7a8c nodeName:}" failed. No retries permitted until 2026-10-30 03:56:50.244737005 +0000 UTC m=+32.073961336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls") pod "dns-default-xkq6m" (UID: "730fcc37-b18e-46e3-aea5-a7fba60b7a8c") : secret "dns-default-metrics-tls" not found
Oct 30 03:56:49.245479 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.245215    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle podName:b752c121-5e44-4df2-9f17-97f0c899516b nodeName:}" failed. No retries permitted until 2026-10-30 03:56:50.24520709 +0000 UTC m=+32.074431421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle") pod "router-default-84bd8c5fd7-885hp" (UID: "b752c121-5e44-4df2-9f17-97f0c899516b") : configmap references non-existent config key: service-ca.crt
Oct 30 03:56:49.245479 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.245247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:49.245479 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.245289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:49.245479 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.245415    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:56:49.245479 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.245452    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs podName:b752c121-5e44-4df2-9f17-97f0c899516b nodeName:}" failed. No retries permitted until 2026-10-30 03:56:50.245440773 +0000 UTC m=+32.074665106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs") pod "router-default-84bd8c5fd7-885hp" (UID: "b752c121-5e44-4df2-9f17-97f0c899516b") : secret "router-metrics-certs-default" not found
Oct 30 03:56:49.245769 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.245505    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:56:49.245769 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.245544    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls podName:788f5892-601d-42cc-b597-528a091feac9 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:50.245533472 +0000 UTC m=+32.074757814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xccsz" (UID: "788f5892-601d-42cc-b597-528a091feac9") : secret "samples-operator-tls" not found
Oct 30 03:56:49.294302 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.294271    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7tc9g"]
Oct 30 03:56:49.295220 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:49.295196    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18cc860_7cd8_48d9_9a7a_40a40a53f379.slice/crio-cea3f6d1877d53e809bcc16713de506133dc4f5201af4ff00915e321b5a6cf17 WatchSource:0}: Error finding container cea3f6d1877d53e809bcc16713de506133dc4f5201af4ff00915e321b5a6cf17: Status 404 returned error can't find the container with id cea3f6d1877d53e809bcc16713de506133dc4f5201af4ff00915e321b5a6cf17
Oct 30 03:56:49.303259 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.303231    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"]
Oct 30 03:56:49.304905 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:49.304884    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c9fb60_7d7b_46b7_8d02_0b51dd1d0221.slice/crio-48d9a1c03192aaa78748b3f27418c14a89cf1cd766a933260e3f9f05b6f6fcdb WatchSource:0}: Error finding container 48d9a1c03192aaa78748b3f27418c14a89cf1cd766a933260e3f9f05b6f6fcdb: Status 404 returned error can't find the container with id 48d9a1c03192aaa78748b3f27418c14a89cf1cd766a933260e3f9f05b6f6fcdb
Oct 30 03:56:49.342459 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.342426    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-d29md"]
Oct 30 03:56:49.345727 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:49.345629    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d40d8b3_4844_4a76_b395_b5b8a2a41438.slice/crio-bd7419982764252cd8990d04642082006bd671ab66a55dbf49c55df7d73e5b0b WatchSource:0}: Error finding container bd7419982764252cd8990d04642082006bd671ab66a55dbf49c55df7d73e5b0b: Status 404 returned error can't find the container with id bd7419982764252cd8990d04642082006bd671ab66a55dbf49c55df7d73e5b0b
Oct 30 03:56:49.345801 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.345759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:49.345912 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.345797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:49.345912 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.345859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:56:49.346051 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.345997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:49.346278 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.346195    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:56:49.346278 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.346249    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls podName:503b82e1-e56c-4c80-9d63-fa8e647c0991 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:50.346230283 +0000 UTC m=+32.175454634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sb794" (UID: "503b82e1-e56c-4c80-9d63-fa8e647c0991") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:56:49.346383 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.346314    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:56:49.346383 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.346326    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbcfd588b-rlkpf: secret "image-registry-tls" not found
Oct 30 03:56:49.346383 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.346364    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls podName:343419d3-210d-4c12-86ff-c92f8f574b27 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:50.346349165 +0000 UTC m=+32.175573500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls") pod "image-registry-6bbcfd588b-rlkpf" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27") : secret "image-registry-tls" not found
Oct 30 03:56:49.346537 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.346426    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:56:49.346537 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.346457    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert podName:01bc8bd9-f593-4acf-97c0-fc6c4beb4867 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:50.346447896 +0000 UTC m=+32.175672225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htskn" (UID: "01bc8bd9-f593-4acf-97c0-fc6c4beb4867") : secret "networking-console-plugin-cert" not found
Oct 30 03:56:49.346537 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.346506    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:56:49.346537 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.346516    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-759fbb47f9-s2s2f: secret "image-registry-tls" not found
Oct 30 03:56:49.346736 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:49.346545    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls podName:395d5cbd-c077-4578-b377-3034e42a2b94 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:50.346534833 +0000 UTC m=+32.175759160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls") pod "image-registry-759fbb47f9-s2s2f" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94") : secret "image-registry-tls" not found
Oct 30 03:56:49.352256 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.351984    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:49.356432 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.356406    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"]
Oct 30 03:56:49.359736 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:49.359711    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe5ff28_0896_4f54_b5cc_5326a0d60752.slice/crio-82fb3af131c173a4d8316136f1fe5412310db2c795ee675e292fc4951e2a4b32 WatchSource:0}: Error finding container 82fb3af131c173a4d8316136f1fe5412310db2c795ee675e292fc4951e2a4b32: Status 404 returned error can't find the container with id 82fb3af131c173a4d8316136f1fe5412310db2c795ee675e292fc4951e2a4b32
Oct 30 03:56:49.379721 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.379688    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-664tl"]
Oct 30 03:56:49.381493 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:49.381469    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907912b8_9313_43b3_b324_e8498ea6519a.slice/crio-9d5d9b77363a73903c75ec607a0ee3521499392ced545186e6663f7bb9b94cc7 WatchSource:0}: Error finding container 9d5d9b77363a73903c75ec607a0ee3521499392ced545186e6663f7bb9b94cc7: Status 404 returned error can't find the container with id 9d5d9b77363a73903c75ec607a0ee3521499392ced545186e6663f7bb9b94cc7
Oct 30 03:56:49.396121 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.396098    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bh464"]
Oct 30 03:56:49.397190 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:49.397161    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f2993a_ed5d_488e_832d_bcba8ddd9706.slice/crio-1064bba8f6bb82f69693e1e393cf229edd2cc40d911a48287ba788f2acc4a59e WatchSource:0}: Error finding container 1064bba8f6bb82f69693e1e393cf229edd2cc40d911a48287ba788f2acc4a59e: Status 404 returned error can't find the container with id 1064bba8f6bb82f69693e1e393cf229edd2cc40d911a48287ba788f2acc4a59e
Oct 30 03:56:49.670789 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.670713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:49.671167 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.670943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:49.671167 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.671105    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:49.687858 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.687830    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 03:56:49.687993 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.687978    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j9mnz\""
Oct 30 03:56:49.688125 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.688104    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 03:56:49.688250 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.688150    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mwxht\""
Oct 30 03:56:49.688250 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.688195    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vnctk\""
Oct 30 03:56:49.910645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.910603    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2" event={"ID":"4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b","Type":"ContainerStarted","Data":"3f6923c74cb8d51b74345c2b62cf915662d2c876238e315d471e4308bfcc82fb"}
Oct 30 03:56:49.918484 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.918451    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-drvrv" event={"ID":"5e68d10c-d217-40d3-945d-c87ccbd7434b","Type":"ContainerStarted","Data":"b1d610cd3112c8c4a8fa9b78308f10aa0b547acf1297c1cc95f149fcdb59c9ae"}
Oct 30 03:56:49.918626 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.918509    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-drvrv" event={"ID":"5e68d10c-d217-40d3-945d-c87ccbd7434b","Type":"ContainerStarted","Data":"c3eb51613da26058dac2cbc5e835f79a7d4ac04cfae6ab2e2184d5017fde4878"}
Oct 30 03:56:49.922849 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.922741    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d" event={"ID":"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221","Type":"ContainerStarted","Data":"48d9a1c03192aaa78748b3f27418c14a89cf1cd766a933260e3f9f05b6f6fcdb"}
Oct 30 03:56:49.924952 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.924924    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jmlk2" event={"ID":"4455e2df-ec71-4748-8920-36b2374af2a1","Type":"ContainerStarted","Data":"2b4413d8da2d90cb9123768dbff1561081c1c330fc038a8cc4d5b30b5bc61cbe"}
Oct 30 03:56:49.925063 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.924975    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jmlk2" event={"ID":"4455e2df-ec71-4748-8920-36b2374af2a1","Type":"ContainerStarted","Data":"a11b2554a66ce72684088750cca0389fa8e7e1900e259271cf46e2db0aeb7acc"}
Oct 30 03:56:49.926553 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.926529    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bh464" event={"ID":"89f2993a-ed5d-488e-832d-bcba8ddd9706","Type":"ContainerStarted","Data":"1064bba8f6bb82f69693e1e393cf229edd2cc40d911a48287ba788f2acc4a59e"}
Oct 30 03:56:49.927952 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.927912    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx" event={"ID":"1fe5ff28-0896-4f54-b5cc-5326a0d60752","Type":"ContainerStarted","Data":"82fb3af131c173a4d8316136f1fe5412310db2c795ee675e292fc4951e2a4b32"}
Oct 30 03:56:49.929461 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.929443    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl" event={"ID":"907912b8-9313-43b3-b324-e8498ea6519a","Type":"ContainerStarted","Data":"9d5d9b77363a73903c75ec607a0ee3521499392ced545186e6663f7bb9b94cc7"}
Oct 30 03:56:49.931212 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.931186    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-d29md" event={"ID":"1d40d8b3-4844-4a76-b395-b5b8a2a41438","Type":"ContainerStarted","Data":"bd7419982764252cd8990d04642082006bd671ab66a55dbf49c55df7d73e5b0b"}
Oct 30 03:56:49.934388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.934315    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7tc9g" event={"ID":"d18cc860-7cd8-48d9-9a7a-40a40a53f379","Type":"ContainerStarted","Data":"a9c8684f4e381ccee521d04387c78ef5d8eebc94f5c497c540978767464476ef"}
Oct 30 03:56:49.934388 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.934362    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7tc9g" event={"ID":"d18cc860-7cd8-48d9-9a7a-40a40a53f379","Type":"ContainerStarted","Data":"cea3f6d1877d53e809bcc16713de506133dc4f5201af4ff00915e321b5a6cf17"}
Oct 30 03:56:49.944292 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.944234    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-trv7j" event={"ID":"d41c0686-14d0-43c4-928e-e38b57021fb7","Type":"ContainerStarted","Data":"92be8bdf88a67f238f28702f1b9f2651bc453f29c06407902f955c491d8d1d9d"}
Oct 30 03:56:49.962471 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:49.962420    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-drvrv" podStartSLOduration=432.962403078 podStartE2EDuration="7m12.962403078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:49.961371457 +0000 UTC m=+31.790595808" watchObservedRunningTime="2026-10-30 03:56:49.962403078 +0000 UTC m=+31.791627431"
Oct 30 03:56:50.006283 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:50.005290    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7tc9g" podStartSLOduration=433.005271913 podStartE2EDuration="7m13.005271913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:50.004480827 +0000 UTC m=+31.833705204" watchObservedRunningTime="2026-10-30 03:56:50.005271913 +0000 UTC m=+31.834496264"
Oct 30 03:56:50.049465 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:50.047674    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jmlk2" podStartSLOduration=433.047656766 podStartE2EDuration="7m13.047656766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:50.047322827 +0000 UTC m=+31.876547178" watchObservedRunningTime="2026-10-30 03:56:50.047656766 +0000 UTC m=+31.876881117"
Oct 30 03:56:50.255822 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:50.254680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:50.255822 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:50.254759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:50.255822 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:50.254838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:50.255822 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:50.254891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:50.255822 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.255018    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:56:50.255822 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.255103    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs podName:b752c121-5e44-4df2-9f17-97f0c899516b nodeName:}" failed. No retries permitted until 2026-10-30 03:56:52.255065555 +0000 UTC m=+34.084289888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs") pod "router-default-84bd8c5fd7-885hp" (UID: "b752c121-5e44-4df2-9f17-97f0c899516b") : secret "router-metrics-certs-default" not found
Oct 30 03:56:50.255822 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.255558    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:56:50.255822 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.255606    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls podName:730fcc37-b18e-46e3-aea5-a7fba60b7a8c nodeName:}" failed. No retries permitted until 2026-10-30 03:56:52.255592095 +0000 UTC m=+34.084816428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls") pod "dns-default-xkq6m" (UID: "730fcc37-b18e-46e3-aea5-a7fba60b7a8c") : secret "dns-default-metrics-tls" not found
Oct 30 03:56:50.255822 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.255690    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle podName:b752c121-5e44-4df2-9f17-97f0c899516b nodeName:}" failed. No retries permitted until 2026-10-30 03:56:52.255665916 +0000 UTC m=+34.084890247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle") pod "router-default-84bd8c5fd7-885hp" (UID: "b752c121-5e44-4df2-9f17-97f0c899516b") : configmap references non-existent config key: service-ca.crt
Oct 30 03:56:50.255822 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.255754    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:56:50.255822 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.255786    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls podName:788f5892-601d-42cc-b597-528a091feac9 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:52.255775546 +0000 UTC m=+34.084999879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xccsz" (UID: "788f5892-601d-42cc-b597-528a091feac9") : secret "samples-operator-tls" not found
Oct 30 03:56:50.355473 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:50.355431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:50.355645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:50.355508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:50.355645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:50.355538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:50.355645 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:50.355594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:56:50.355840 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.355808    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:56:50.355895 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.355867    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert podName:01bc8bd9-f593-4acf-97c0-fc6c4beb4867 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:52.355850487 +0000 UTC m=+34.185074816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htskn" (UID: "01bc8bd9-f593-4acf-97c0-fc6c4beb4867") : secret "networking-console-plugin-cert" not found
Oct 30 03:56:50.356271 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.356247    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:56:50.356271 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.356270    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-759fbb47f9-s2s2f: secret "image-registry-tls" not found
Oct 30 03:56:50.356469 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.356317    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls podName:395d5cbd-c077-4578-b377-3034e42a2b94 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:52.356302483 +0000 UTC m=+34.185526813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls") pod "image-registry-759fbb47f9-s2s2f" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94") : secret "image-registry-tls" not found
Oct 30 03:56:50.356469 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.356391    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:56:50.356469 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.356401    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbcfd588b-rlkpf: secret "image-registry-tls" not found
Oct 30 03:56:50.356469 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.356433    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls podName:343419d3-210d-4c12-86ff-c92f8f574b27 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:52.356421226 +0000 UTC m=+34.185645559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls") pod "image-registry-6bbcfd588b-rlkpf" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27") : secret "image-registry-tls" not found
Oct 30 03:56:50.356684 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.356486    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:56:50.356684 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:50.356517    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls podName:503b82e1-e56c-4c80-9d63-fa8e647c0991 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:52.356507504 +0000 UTC m=+34.185731838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sb794" (UID: "503b82e1-e56c-4c80-9d63-fa8e647c0991") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:56:51.962534 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:51.961668    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2" event={"ID":"4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b","Type":"ContainerStarted","Data":"e6d266c529ab6c8f9c068b9dd3c5e8283f9917050cc8c40c45d4711583dc9282"}
Oct 30 03:56:52.005763 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.005534    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2" podStartSLOduration=424.101733787 podStartE2EDuration="7m6.005519906s" totalImagesPullingTime="1.903786119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:52.004425438 +0000 UTC m=+33.833649811" watchObservedRunningTime="2026-10-30 03:56:52.005519906 +0000 UTC m=+33.834744255"
Oct 30 03:56:52.246006 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.245953    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:52.278318 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.278281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:52.278489 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.278404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:52.278489 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.278465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:52.278597 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.278525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:52.278647 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.278638    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:56:52.278700 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.278687    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls podName:788f5892-601d-42cc-b597-528a091feac9 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:56.278672897 +0000 UTC m=+38.107897224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xccsz" (UID: "788f5892-601d-42cc-b597-528a091feac9") : secret "samples-operator-tls" not found
Oct 30 03:56:52.279039 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.279021    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:56:52.279143 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.279071    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs podName:b752c121-5e44-4df2-9f17-97f0c899516b nodeName:}" failed. No retries permitted until 2026-10-30 03:56:56.279057029 +0000 UTC m=+38.108281375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs") pod "router-default-84bd8c5fd7-885hp" (UID: "b752c121-5e44-4df2-9f17-97f0c899516b") : secret "router-metrics-certs-default" not found
Oct 30 03:56:52.279143 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.279132    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:56:52.279243 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.279165    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls podName:730fcc37-b18e-46e3-aea5-a7fba60b7a8c nodeName:}" failed. No retries permitted until 2026-10-30 03:56:56.279152786 +0000 UTC m=+38.108377116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls") pod "dns-default-xkq6m" (UID: "730fcc37-b18e-46e3-aea5-a7fba60b7a8c") : secret "dns-default-metrics-tls" not found
Oct 30 03:56:52.279243 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.279196    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle podName:b752c121-5e44-4df2-9f17-97f0c899516b nodeName:}" failed. No retries permitted until 2026-10-30 03:56:56.279187014 +0000 UTC m=+38.108411348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle") pod "router-default-84bd8c5fd7-885hp" (UID: "b752c121-5e44-4df2-9f17-97f0c899516b") : configmap references non-existent config key: service-ca.crt
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.379576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.379660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.379690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.379737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.379964    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.379982    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-759fbb47f9-s2s2f: secret "image-registry-tls" not found
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.380030    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls podName:395d5cbd-c077-4578-b377-3034e42a2b94 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:56.380016173 +0000 UTC m=+38.209240499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls") pod "image-registry-759fbb47f9-s2s2f" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94") : secret "image-registry-tls" not found
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.380345    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.380356    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbcfd588b-rlkpf: secret "image-registry-tls" not found
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.380400    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls podName:343419d3-210d-4c12-86ff-c92f8f574b27 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:56.380387111 +0000 UTC m=+38.209611439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls") pod "image-registry-6bbcfd588b-rlkpf" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27") : secret "image-registry-tls" not found
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.380452    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.380475    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls podName:503b82e1-e56c-4c80-9d63-fa8e647c0991 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:56.380468864 +0000 UTC m=+38.209693191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sb794" (UID: "503b82e1-e56c-4c80-9d63-fa8e647c0991") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.380511    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:56:52.380570 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.380531    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert podName:01bc8bd9-f593-4acf-97c0-fc6c4beb4867 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:56.380524464 +0000 UTC m=+38.209748790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htskn" (UID: "01bc8bd9-f593-4acf-97c0-fc6c4beb4867") : secret "networking-console-plugin-cert" not found
Oct 30 03:56:52.478815 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.478734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:52.480499 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.480469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:52.480617 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.480556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:56:52.480843 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.480821    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 03:56:52.480906 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:52.480881    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs podName:623e4b93-06e4-44f9-bd85-d9b663df4834 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:24.480863832 +0000 UTC m=+66.310088167 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs") pod "network-metrics-daemon-b67qk" (UID: "623e4b93-06e4-44f9-bd85-d9b663df4834") : secret "metrics-daemon-secret" not found
Oct 30 03:56:52.484553 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.484533    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:52.582212 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.581485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8xt\" (UniqueName: \"kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt\") pod \"network-check-target-k2wgk\" (UID: \"4c692538-d934-46cf-8c4c-d6a888dd00a1\") " pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:52.585102 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.585035    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr8xt\" (UniqueName: \"kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt\") pod \"network-check-target-k2wgk\" (UID: \"4c692538-d934-46cf-8c4c-d6a888dd00a1\") " pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:52.725920 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.725882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:52.741452 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:52.741383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:56:53.084420 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:53.084353    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k2wgk"]
Oct 30 03:56:53.085774 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:53.085731    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c692538_d934_46cf_8c4c_d6a888dd00a1.slice/crio-f2c832f8ad8bb903525ec645e2610c91ec3478cc3f73deca60f4b1ba28e3ec15 WatchSource:0}: Error finding container f2c832f8ad8bb903525ec645e2610c91ec3478cc3f73deca60f4b1ba28e3ec15: Status 404 returned error can't find the container with id f2c832f8ad8bb903525ec645e2610c91ec3478cc3f73deca60f4b1ba28e3ec15
Oct 30 03:56:53.118360 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:53.118281    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d67gd"]
Oct 30 03:56:53.122704 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:53.122671    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9a426a_587d_4611_9298_90fc4b4f1dc3.slice/crio-692e3aaa48a317e6b6d35c21fae3e7f30a86da5907eef5cb35b48d4630d3e0f7 WatchSource:0}: Error finding container 692e3aaa48a317e6b6d35c21fae3e7f30a86da5907eef5cb35b48d4630d3e0f7: Status 404 returned error can't find the container with id 692e3aaa48a317e6b6d35c21fae3e7f30a86da5907eef5cb35b48d4630d3e0f7
Oct 30 03:56:53.970109 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:53.970063    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/0.log"
Oct 30 03:56:53.970258 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:53.970129    2577 generic.go:356] "Generic (PLEG): container finished" podID="907912b8-9313-43b3-b324-e8498ea6519a" containerID="c6df8669da3210ac7ab498bdfc31a20a97dc616f446ddf8c645e4f2299762c8d" exitCode=255
Oct 30 03:56:53.970434 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:53.970356    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl" event={"ID":"907912b8-9313-43b3-b324-e8498ea6519a","Type":"ContainerDied","Data":"c6df8669da3210ac7ab498bdfc31a20a97dc616f446ddf8c645e4f2299762c8d"}
Oct 30 03:56:53.970552 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:53.970503    2577 scope.go:122] "RemoveContainer" containerID="c6df8669da3210ac7ab498bdfc31a20a97dc616f446ddf8c645e4f2299762c8d"
Oct 30 03:56:53.972553 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:53.972509    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-trv7j" event={"ID":"d41c0686-14d0-43c4-928e-e38b57021fb7","Type":"ContainerStarted","Data":"be3dd1a8d7dd162c3aa399e155ea9ae869da5a735b4514fc44940e03253c4c21"}
Oct 30 03:56:53.974593 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:53.974565    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d67gd" event={"ID":"bc9a426a-587d-4611-9298-90fc4b4f1dc3","Type":"ContainerStarted","Data":"692e3aaa48a317e6b6d35c21fae3e7f30a86da5907eef5cb35b48d4630d3e0f7"}
Oct 30 03:56:53.976568 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:53.976516    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k2wgk" event={"ID":"4c692538-d934-46cf-8c4c-d6a888dd00a1","Type":"ContainerStarted","Data":"f2c832f8ad8bb903525ec645e2610c91ec3478cc3f73deca60f4b1ba28e3ec15"}
Oct 30 03:56:54.060333 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.060280    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-trv7j" podStartSLOduration=423.64147528 podStartE2EDuration="7m8.06026502s" totalImagesPullingTime="4.41878974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:54.059633802 +0000 UTC m=+35.888858150" watchObservedRunningTime="2026-10-30 03:56:54.06026502 +0000 UTC m=+35.889489369"
Oct 30 03:56:54.469434 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.469411    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7x92g"]
Oct 30 03:56:54.503274 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.502739    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7x92g"]
Oct 30 03:56:54.503274 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.502872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.519904 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.519752    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 03:56:54.519904 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.519836    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-846td\""
Oct 30 03:56:54.519904 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.519884    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 03:56:54.550631 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.550605    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-qddvs-q47fm"]
Oct 30 03:56:54.574245 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.571368    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.584270 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.584208    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-qddvs-q47fm"]
Oct 30 03:56:54.605838 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.604792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d2109e3-d762-4bf9-bf3a-d405148ec32c-crio-socket\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.605838 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.604852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/67dad9d6-9671-49f3-84a2-024463451062-archives-path\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.605838 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.604886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.605838 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.604931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.605838 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.604969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-service-ca-bundle\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.605838 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.605069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-trusted-ca-bundle\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.605838 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.605125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq4h9\" (UniqueName: \"kubernetes.io/projected/67dad9d6-9671-49f3-84a2-024463451062-kube-api-access-xq4h9\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.605838 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.605308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d2109e3-d762-4bf9-bf3a-d405148ec32c-data-volume\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.605838 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.605354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9hcr\" (UniqueName: \"kubernetes.io/projected/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-api-access-s9hcr\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.620298 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.620258    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xpsj8\""
Oct 30 03:56:54.663392 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.663023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:54.706417 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.706391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-trusted-ca-bundle\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.706569 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.706430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq4h9\" (UniqueName: \"kubernetes.io/projected/67dad9d6-9671-49f3-84a2-024463451062-kube-api-access-xq4h9\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.706569 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.706479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d2109e3-d762-4bf9-bf3a-d405148ec32c-data-volume\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.706569 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.706505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9hcr\" (UniqueName: \"kubernetes.io/projected/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-api-access-s9hcr\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.706569 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.706538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d2109e3-d762-4bf9-bf3a-d405148ec32c-crio-socket\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.706754 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.706575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/67dad9d6-9671-49f3-84a2-024463451062-archives-path\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.706754 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.706594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.706754 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.706631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.706754 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.706654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-service-ca-bundle\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.707951 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.707166    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-service-ca-bundle\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.707951 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:54.707274    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:56:54.707951 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:54.707338    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls podName:0d2109e3-d762-4bf9-bf3a-d405148ec32c nodeName:}" failed. No retries permitted until 2026-10-30 03:56:55.207314309 +0000 UTC m=+37.036538652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7x92g" (UID: "0d2109e3-d762-4bf9-bf3a-d405148ec32c") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:56:54.707951 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.707488    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/67dad9d6-9671-49f3-84a2-024463451062-archives-path\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.707951 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:54.707533    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:56:54.707951 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.707544    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d2109e3-d762-4bf9-bf3a-d405148ec32c-data-volume\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.707951 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:54.707569    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm podName:0d2109e3-d762-4bf9-bf3a-d405148ec32c nodeName:}" failed. No retries permitted until 2026-10-30 03:56:55.207558244 +0000 UTC m=+37.036782571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7x92g" (UID: "0d2109e3-d762-4bf9-bf3a-d405148ec32c") : configmap "kube-rbac-proxy" not found
Oct 30 03:56:54.707951 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.707576    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d2109e3-d762-4bf9-bf3a-d405148ec32c-crio-socket\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.707951 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.707936    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-trusted-ca-bundle\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.734623 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.734575    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9hcr\" (UniqueName: \"kubernetes.io/projected/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-api-access-s9hcr\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:54.736743 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.736707    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq4h9\" (UniqueName: \"kubernetes.io/projected/67dad9d6-9671-49f3-84a2-024463451062-kube-api-access-xq4h9\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.892046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.891978    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:54.922638 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.922407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:56:54.985930 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.985714    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d" event={"ID":"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221","Type":"ContainerStarted","Data":"0dbc6894e356c47ef9edfea13013effc603875c6104687b31a0dd0e9a583963e"}
Oct 30 03:56:54.991636 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.991067    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bh464" event={"ID":"89f2993a-ed5d-488e-832d-bcba8ddd9706","Type":"ContainerStarted","Data":"e825efafc0a8c3f42f376dd4fc9521894a90e2b33d873e0f4aea44aab677b0e8"}
Oct 30 03:56:54.995983 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.995928    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx" event={"ID":"1fe5ff28-0896-4f54-b5cc-5326a0d60752","Type":"ContainerStarted","Data":"50ca62ec3fb587bd22466f206da0132db84f9ea73e44cd802dbd232beadf7577"}
Oct 30 03:56:54.999911 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:54.999648    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 03:56:55.000880 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.000308    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/0.log"
Oct 30 03:56:55.000880 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.000344    2577 generic.go:356] "Generic (PLEG): container finished" podID="907912b8-9313-43b3-b324-e8498ea6519a" containerID="9bb9bd5f4ffffda1ea02732ce99b0847014da4bd97bc3c7228c3d5abeb0a2e15" exitCode=255
Oct 30 03:56:55.000880 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.000576    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl" event={"ID":"907912b8-9313-43b3-b324-e8498ea6519a","Type":"ContainerDied","Data":"9bb9bd5f4ffffda1ea02732ce99b0847014da4bd97bc3c7228c3d5abeb0a2e15"}
Oct 30 03:56:55.000880 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.000645    2577 scope.go:122] "RemoveContainer" containerID="c6df8669da3210ac7ab498bdfc31a20a97dc616f446ddf8c645e4f2299762c8d"
Oct 30 03:56:55.000880 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.000665    2577 scope.go:122] "RemoveContainer" containerID="9bb9bd5f4ffffda1ea02732ce99b0847014da4bd97bc3c7228c3d5abeb0a2e15"
Oct 30 03:56:55.000880 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:55.000828    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-664tl_openshift-console-operator(907912b8-9313-43b3-b324-e8498ea6519a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl" podUID="907912b8-9313-43b3-b324-e8498ea6519a"
Oct 30 03:56:55.005475 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.004977    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-d29md" event={"ID":"1d40d8b3-4844-4a76-b395-b5b8a2a41438","Type":"ContainerStarted","Data":"5ff9ff5a567a9d13a5ba8b79b97fe0a50feecde34204848cd75854d276073522"}
Oct 30 03:56:55.032309 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.031589    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d" podStartSLOduration=422.903594374 podStartE2EDuration="7m8.03157522s" totalImagesPullingTime="5.127980846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:55.030342857 +0000 UTC m=+36.859567220" watchObservedRunningTime="2026-10-30 03:56:55.03157522 +0000 UTC m=+36.860799570"
Oct 30 03:56:55.125016 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.124313    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx" podStartSLOduration=422.051462284 podStartE2EDuration="7m7.124298802s" totalImagesPullingTime="5.072836518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:55.123969632 +0000 UTC m=+36.953193991" watchObservedRunningTime="2026-10-30 03:56:55.124298802 +0000 UTC m=+36.953523150"
Oct 30 03:56:55.213030 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.212240    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-d29md" podStartSLOduration=391.71986857 podStartE2EDuration="6m37.212222696s" totalImagesPullingTime="5.492354126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:55.168227602 +0000 UTC m=+36.997451953" watchObservedRunningTime="2026-10-30 03:56:55.212222696 +0000 UTC m=+37.041447048"
Oct 30 03:56:55.213030 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.212854    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bh464" podStartSLOduration=2.156245005 podStartE2EDuration="7.212844843s" totalImagesPullingTime="5.056599838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:55.211054788 +0000 UTC m=+37.040279137" watchObservedRunningTime="2026-10-30 03:56:55.212844843 +0000 UTC m=+37.042069213"
Oct 30 03:56:55.213759 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.213735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:55.213880 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.213773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:55.213880 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:55.213869    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:56:55.215121 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:55.213915    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm podName:0d2109e3-d762-4bf9-bf3a-d405148ec32c nodeName:}" failed. No retries permitted until 2026-10-30 03:56:56.21390426 +0000 UTC m=+38.043128587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7x92g" (UID: "0d2109e3-d762-4bf9-bf3a-d405148ec32c") : configmap "kube-rbac-proxy" not found
Oct 30 03:56:55.215121 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:55.213987    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:56:55.215121 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:55.214023    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls podName:0d2109e3-d762-4bf9-bf3a-d405148ec32c nodeName:}" failed. No retries permitted until 2026-10-30 03:56:56.214011502 +0000 UTC m=+38.043235831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7x92g" (UID: "0d2109e3-d762-4bf9-bf3a-d405148ec32c") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:56:55.256253 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.256220    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-qddvs-q47fm"]
Oct 30 03:56:55.683266 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.681946    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk"]
Oct 30 03:56:55.703108 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.702359    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7"]
Oct 30 03:56:55.703108 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.702490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk"
Oct 30 03:56:55.718604 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.718579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2qhn\" (UniqueName: \"kubernetes.io/projected/491c21e4-c0e9-4d60-a00e-c896bcda29f7-kube-api-access-m2qhn\") pod \"migrator-66df5d496b-96bwk\" (UID: \"491c21e4-c0e9-4d60-a00e-c896bcda29f7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk"
Oct 30 03:56:55.726434 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.726402    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tsnxp\""
Oct 30 03:56:55.728298 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.728188    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk"]
Oct 30 03:56:55.728298 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.728228    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7"]
Oct 30 03:56:55.728630 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.728608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7"
Oct 30 03:56:55.758340 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.758025    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:55.808174 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.808125    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 03:56:55.808330 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.808193    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 03:56:55.820055 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.820025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jq7h\" (UniqueName: \"kubernetes.io/projected/5a290674-68db-4847-b2a1-ad968844f699-kube-api-access-6jq7h\") pod \"migrator-66df5d496b-2nlx7\" (UID: \"5a290674-68db-4847-b2a1-ad968844f699\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7"
Oct 30 03:56:55.820291 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.820262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2qhn\" (UniqueName: \"kubernetes.io/projected/491c21e4-c0e9-4d60-a00e-c896bcda29f7-kube-api-access-m2qhn\") pod \"migrator-66df5d496b-96bwk\" (UID: \"491c21e4-c0e9-4d60-a00e-c896bcda29f7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk"
Oct 30 03:56:55.846392 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.846362    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2qhn\" (UniqueName: \"kubernetes.io/projected/491c21e4-c0e9-4d60-a00e-c896bcda29f7-kube-api-access-m2qhn\") pod \"migrator-66df5d496b-96bwk\" (UID: \"491c21e4-c0e9-4d60-a00e-c896bcda29f7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk"
Oct 30 03:56:55.921030 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.920996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jq7h\" (UniqueName: \"kubernetes.io/projected/5a290674-68db-4847-b2a1-ad968844f699-kube-api-access-6jq7h\") pod \"migrator-66df5d496b-2nlx7\" (UID: \"5a290674-68db-4847-b2a1-ad968844f699\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7"
Oct 30 03:56:55.946804 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:55.946733    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jq7h\" (UniqueName: \"kubernetes.io/projected/5a290674-68db-4847-b2a1-ad968844f699-kube-api-access-6jq7h\") pod \"migrator-66df5d496b-2nlx7\" (UID: \"5a290674-68db-4847-b2a1-ad968844f699\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7"
Oct 30 03:56:56.011120 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.011038    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qddvs-q47fm" event={"ID":"67dad9d6-9671-49f3-84a2-024463451062","Type":"ContainerStarted","Data":"e97abdeeb6ea2d5d5c7f770caf73c93d7fcd2711bf71f3f4d25ae5460d6b71f8"}
Oct 30 03:56:56.011120 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.011124    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qddvs-q47fm" event={"ID":"67dad9d6-9671-49f3-84a2-024463451062","Type":"ContainerStarted","Data":"d0353988f43af1839855daa5855f75647988e0f6163e7790e44cd475190d00eb"}
Oct 30 03:56:56.013253 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.013218    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 03:56:56.013855 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.013828    2577 scope.go:122] "RemoveContainer" containerID="9bb9bd5f4ffffda1ea02732ce99b0847014da4bd97bc3c7228c3d5abeb0a2e15"
Oct 30 03:56:56.014002 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.013961    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-664tl_openshift-console-operator(907912b8-9313-43b3-b324-e8498ea6519a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl" podUID="907912b8-9313-43b3-b324-e8498ea6519a"
Oct 30 03:56:56.038876 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.038829    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk"
Oct 30 03:56:56.039900 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.039872    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:56.056795 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.056736    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-qddvs-q47fm" podStartSLOduration=2.056718467 podStartE2EDuration="2.056718467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:56.055205536 +0000 UTC m=+37.884429885" watchObservedRunningTime="2026-10-30 03:56:56.056718467 +0000 UTC m=+37.885942818"
Oct 30 03:56:56.061569 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.061549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7"
Oct 30 03:56:56.158509 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.158483    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qc52w"]
Oct 30 03:56:56.177133 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.177100    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:56:56.180599 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.180575    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qc52w"]
Oct 30 03:56:56.193927 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.193891    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-n6x98\""
Oct 30 03:56:56.194051 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.193910    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 03:56:56.194051 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.193941    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 03:56:56.194051 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.193998    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 03:56:56.194174 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.194057    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 03:56:56.226460 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.226404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-signing-key\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:56:56.226460 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.226450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-signing-cabundle\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:56:56.226759 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.226737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mlbm\" (UniqueName: \"kubernetes.io/projected/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-kube-api-access-6mlbm\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:56:56.226817 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.226790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:56.226859 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.226839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:56.226957 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.226937    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:56:56.227021 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.227010    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm podName:0d2109e3-d762-4bf9-bf3a-d405148ec32c nodeName:}" failed. No retries permitted until 2026-10-30 03:56:58.226991313 +0000 UTC m=+40.056215644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7x92g" (UID: "0d2109e3-d762-4bf9-bf3a-d405148ec32c") : configmap "kube-rbac-proxy" not found
Oct 30 03:56:56.227102 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.227026    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:56:56.227102 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.227097    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls podName:0d2109e3-d762-4bf9-bf3a-d405148ec32c nodeName:}" failed. No retries permitted until 2026-10-30 03:56:58.227057884 +0000 UTC m=+40.056282216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7x92g" (UID: "0d2109e3-d762-4bf9-bf3a-d405148ec32c") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:56:56.327532 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.327504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:56:56.327638 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.327573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-signing-key\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:56:56.327638 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.327599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-signing-cabundle\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:56:56.327638 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.327627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:56.327789 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.327658    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:56:56.327789 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.327724    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls podName:788f5892-601d-42cc-b597-528a091feac9 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:04.327705058 +0000 UTC m=+46.156929391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xccsz" (UID: "788f5892-601d-42cc-b597-528a091feac9") : secret "samples-operator-tls" not found
Oct 30 03:56:56.327789 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.327748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:56:56.327789 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.327770    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:56:56.327972 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.327801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlbm\" (UniqueName: \"kubernetes.io/projected/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-kube-api-access-6mlbm\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:56:56.327972 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.327836    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs podName:b752c121-5e44-4df2-9f17-97f0c899516b nodeName:}" failed. No retries permitted until 2026-10-30 03:57:04.327803902 +0000 UTC m=+46.157028231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs") pod "router-default-84bd8c5fd7-885hp" (UID: "b752c121-5e44-4df2-9f17-97f0c899516b") : secret "router-metrics-certs-default" not found
Oct 30 03:56:56.327972 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.327898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:56:56.328153 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.328027    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle podName:b752c121-5e44-4df2-9f17-97f0c899516b nodeName:}" failed. No retries permitted until 2026-10-30 03:57:04.328016509 +0000 UTC m=+46.157240837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle") pod "router-default-84bd8c5fd7-885hp" (UID: "b752c121-5e44-4df2-9f17-97f0c899516b") : configmap references non-existent config key: service-ca.crt
Oct 30 03:56:56.328153 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.328127    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:56:56.328252 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.328169    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls podName:730fcc37-b18e-46e3-aea5-a7fba60b7a8c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:04.328154418 +0000 UTC m=+46.157378761 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls") pod "dns-default-xkq6m" (UID: "730fcc37-b18e-46e3-aea5-a7fba60b7a8c") : secret "dns-default-metrics-tls" not found
Oct 30 03:56:56.328620 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.328595    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-signing-cabundle\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:56:56.330803 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.330780    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-signing-key\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:56:56.352685 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.352645    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlbm\" (UniqueName: \"kubernetes.io/projected/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-kube-api-access-6mlbm\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:56:56.377428 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:56.377400    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491c21e4_c0e9_4d60_a00e_c896bcda29f7.slice/crio-3d10c03b71d397c3dd99473e04d2d75934fdf83895543ead219e640e01653439 WatchSource:0}: Error finding container 3d10c03b71d397c3dd99473e04d2d75934fdf83895543ead219e640e01653439: Status 404 returned error can't find the container with id 3d10c03b71d397c3dd99473e04d2d75934fdf83895543ead219e640e01653439
Oct 30 03:56:56.378295 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.378254    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk"]
Oct 30 03:56:56.387708 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.387687    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7"]
Oct 30 03:56:56.388431 ip-10-0-139-99 kubenswrapper[2577]: W1030 03:56:56.388408    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a290674_68db_4847_b2a1_ad968844f699.slice/crio-bf454773ec65875e68dd75c8c21151b3f7f117b638fdbb69e91cbdc7770d4cd6 WatchSource:0}: Error finding container bf454773ec65875e68dd75c8c21151b3f7f117b638fdbb69e91cbdc7770d4cd6: Status 404 returned error can't find the container with id bf454773ec65875e68dd75c8c21151b3f7f117b638fdbb69e91cbdc7770d4cd6
Oct 30 03:56:56.428323 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.428293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:56:56.428458 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.428443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:56:56.428523 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.428461    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:56:56.428523 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.428497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:56:56.428523 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.428521    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert podName:01bc8bd9-f593-4acf-97c0-fc6c4beb4867 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:04.42850187 +0000 UTC m=+46.257726203 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htskn" (UID: "01bc8bd9-f593-4acf-97c0-fc6c4beb4867") : secret "networking-console-plugin-cert" not found
Oct 30 03:56:56.428683 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.428564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:56:56.428683 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.428579    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:56:56.428683 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.428580    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:56:56.428683 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.428593    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbcfd588b-rlkpf: secret "image-registry-tls" not found
Oct 30 03:56:56.428683 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.428597    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-759fbb47f9-s2s2f: secret "image-registry-tls" not found
Oct 30 03:56:56.428683 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.428634    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls podName:395d5cbd-c077-4578-b377-3034e42a2b94 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:04.42861988 +0000 UTC m=+46.257844209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls") pod "image-registry-759fbb47f9-s2s2f" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94") : secret "image-registry-tls" not found
Oct 30 03:56:56.428683 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.428647    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls podName:343419d3-210d-4c12-86ff-c92f8f574b27 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:04.428641517 +0000 UTC m=+46.257865844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls") pod "image-registry-6bbcfd588b-rlkpf" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27") : secret "image-registry-tls" not found
Oct 30 03:56:56.428683 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.428672    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:56:56.429016 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:56.428721    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls podName:503b82e1-e56c-4c80-9d63-fa8e647c0991 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:04.428707984 +0000 UTC m=+46.257932316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sb794" (UID: "503b82e1-e56c-4c80-9d63-fa8e647c0991") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:56:56.500651 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.500596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:56:56.691361 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.691282    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:56.814604 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.814570    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qc52w"]
Oct 30 03:56:56.909863 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:56.909838    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:57.020477 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:57.020385    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7" event={"ID":"5a290674-68db-4847-b2a1-ad968844f699","Type":"ContainerStarted","Data":"bf454773ec65875e68dd75c8c21151b3f7f117b638fdbb69e91cbdc7770d4cd6"}
Oct 30 03:56:57.024964 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:57.024853    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k2wgk" event={"ID":"4c692538-d934-46cf-8c4c-d6a888dd00a1","Type":"ContainerStarted","Data":"ca7c3fabe83984cec912f211b2856eb703c478cf3e979e0d65cf59a2182a3bfb"}
Oct 30 03:56:57.024964 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:57.024932    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:56:57.026952 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:57.026930    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w" event={"ID":"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67","Type":"ContainerStarted","Data":"e7562a0a517caeafe6f6aac15ee82e73d35581761732f72efdf36f5f5ef4f72e"}
Oct 30 03:56:57.027094 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:57.026971    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w" event={"ID":"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67","Type":"ContainerStarted","Data":"23096442889c9f8710b74a793b9bc069ff96edf5d29beb254ad357e11d50f724"}
Oct 30 03:56:57.029090 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:57.029021    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk" event={"ID":"491c21e4-c0e9-4d60-a00e-c896bcda29f7","Type":"ContainerStarted","Data":"3d10c03b71d397c3dd99473e04d2d75934fdf83895543ead219e640e01653439"}
Oct 30 03:56:57.069419 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:57.069331    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k2wgk" podStartSLOduration=35.827068601 podStartE2EDuration="39.069313128s" totalImagesPullingTime="3.242244527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:57.06862957 +0000 UTC m=+38.897853918" watchObservedRunningTime="2026-10-30 03:56:57.069313128 +0000 UTC m=+38.898537470"
Oct 30 03:56:57.631984 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:57.631954    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6njd4_b555cf82-5423-4c47-a004-ae399654683b/dns-node-resolver/0.log"
Oct 30 03:56:57.673571 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:57.673533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:57.896616 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:57.896534    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:58.123576 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:58.123543    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:58.232650 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:58.232608    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mc77t_0384e218-1937-4099-8566-67775cda0521/node-ca/0.log"
Oct 30 03:56:58.249647 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:58.248529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:58.249647 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:58.248583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:56:58.249647 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:58.248706    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:56:58.249647 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:58.248757    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm podName:0d2109e3-d762-4bf9-bf3a-d405148ec32c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:02.248740296 +0000 UTC m=+44.077964628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7x92g" (UID: "0d2109e3-d762-4bf9-bf3a-d405148ec32c") : configmap "kube-rbac-proxy" not found
Oct 30 03:56:58.249647 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:58.249164    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:56:58.249647 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:58.249217    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls podName:0d2109e3-d762-4bf9-bf3a-d405148ec32c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:02.249201101 +0000 UTC m=+44.078425437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7x92g" (UID: "0d2109e3-d762-4bf9-bf3a-d405148ec32c") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:56:58.357174 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:58.357143    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:58.638418 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:58.638336    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bh464_89f2993a-ed5d-488e-832d-bcba8ddd9706/serve-healthcheck-canary/0.log"
Oct 30 03:56:58.878212 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:58.878181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:58.930986 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:58.930907    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:58.930986 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:58.930967    2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:56:58.931462 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:58.931348    2577 scope.go:122] "RemoveContainer" containerID="9bb9bd5f4ffffda1ea02732ce99b0847014da4bd97bc3c7228c3d5abeb0a2e15"
Oct 30 03:56:58.931559 ip-10-0-139-99 kubenswrapper[2577]: E1030 03:56:58.931539    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-664tl_openshift-console-operator(907912b8-9313-43b3-b324-e8498ea6519a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl" podUID="907912b8-9313-43b3-b324-e8498ea6519a"
Oct 30 03:56:59.052521 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:59.052486    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7" event={"ID":"5a290674-68db-4847-b2a1-ad968844f699","Type":"ContainerStarted","Data":"84d462714240e44e6ca3ab7cd2e6f215810ee05a9d04d14c9367aa4ff6c4d8ea"}
Oct 30 03:56:59.054331 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:59.054300    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk" event={"ID":"491c21e4-c0e9-4d60-a00e-c896bcda29f7","Type":"ContainerStarted","Data":"1536e504851687676b9d2b1aeacca5d172dad60b45ea58c0274db2959e009a4e"}
Oct 30 03:56:59.097046 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:59.097003    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7" podStartSLOduration=2.296215519 podStartE2EDuration="4.096986985s" totalImagesPullingTime="1.800771466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:59.096529691 +0000 UTC m=+40.925754041" watchObservedRunningTime="2026-10-30 03:56:59.096986985 +0000 UTC m=+40.926211368"
Oct 30 03:56:59.098147 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:59.098117    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:59.098411 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:59.098381    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w" podStartSLOduration=3.09837109 podStartE2EDuration="3.09837109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:57.113249294 +0000 UTC m=+38.942473666" watchObservedRunningTime="2026-10-30 03:56:59.09837109 +0000 UTC m=+40.927595438"
Oct 30 03:56:59.139737 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:59.139702    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk" podStartSLOduration=2.32554951 podStartE2EDuration="4.139690972s" totalImagesPullingTime="1.814141462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:59.138971388 +0000 UTC m=+40.968195739" watchObservedRunningTime="2026-10-30 03:56:59.139690972 +0000 UTC m=+40.968915321"
Oct 30 03:56:59.234366 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:59.234336    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx_1fe5ff28-0896-4f54-b5cc-5326a0d60752/kube-storage-version-migrator-operator/0.log"
Oct 30 03:56:59.545462 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:59.545384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:59.771778 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:59.771744    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:59.992976 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:56:59.992954    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:00.059372 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:57:00.059286    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d67gd" event={"ID":"bc9a426a-587d-4611-9298-90fc4b4f1dc3","Type":"ContainerStarted","Data":"79199b004d2849ffd8d9b6eb12411dbbcb09aa8e58fb7ec2fff15c6b21da7c6e"}
Oct 30 03:57:00.103731 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:57:00.103683    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d67gd" podStartSLOduration=34.320714836 podStartE2EDuration="41.103669954s" totalImagesPullingTime="6.782955118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:00.10292672 +0000 UTC m=+41.932151070" watchObservedRunningTime="2026-10-30 03:57:00.103669954 +0000 UTC m=+41.932894302"
Oct 30 03:57:00.204029 ip-10-0-139-99 kubenswrapper[2577]: I1030 03:57:00.203944    2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:57:00.204179 ip-10-0-139-99 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 03:57:00.214459 ip-10-0-139-99 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 03:57:00.214631 ip-10-0-139-99 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 03:57:00.214679 ip-10-0-139-99 systemd[1]: kubelet.service: Consumed 3.211s CPU time.
Oct 30 03:57:00.227521 ip-10-0-139-99 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 03:57:00.314936 ip-10-0-139-99 kubenswrapper[5072]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:57:00.314936 ip-10-0-139-99 kubenswrapper[5072]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 03:57:00.314936 ip-10-0-139-99 kubenswrapper[5072]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:57:00.314936 ip-10-0-139-99 kubenswrapper[5072]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:57:00.317058 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.316979    5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:00.317058 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317058    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317063    5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317068    5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317071    5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317089    5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317092    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317094    5072 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317097    5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317100    5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317102    5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317105    5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317108    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317111    5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317113    5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317116    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317118    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317121    5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317124    5072 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317126    5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317129    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:00.317152 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317132    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317134    5072 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317137    5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317139    5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317142    5072 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317145    5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317156    5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317159    5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317162    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317165    5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317168    5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317170    5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317173    5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317175    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317178    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317180    5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317183    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317185    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317188    5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:00.317630 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317190    5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317193    5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317195    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317198    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317201    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317203    5072 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317205    5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317208    5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317211    5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317219    5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317221    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317225    5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317228    5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317230    5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317235    5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317238    5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317242    5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317245    5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317248    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:00.318124 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317250    5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317258    5072 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317261    5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317264    5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317266    5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317269    5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317272    5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317275    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317277    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317279    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317282    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317285    5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317288    5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317291    5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317294    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317296    5072 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317299    5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317301    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317304    5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317306    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:00.318605 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317309    5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317311    5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317314    5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317316    5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317319    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317321    5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317323    5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317326    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317328    5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317330    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317333    5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317335    5072 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317337    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317340    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317347    5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317350    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317353    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317355    5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317360    5072 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317362    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:00.319110 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317365    5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317368    5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317370    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317373    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317375    5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317378    5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.317381    5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318175    5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318182    5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318185    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318188    5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318191    5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318193    5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318196    5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318199    5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318201    5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318204    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318207    5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318209    5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318212    5072 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:00.319588 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318214    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318217    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318220    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318222    5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318225    5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318228    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318230    5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318234    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318237    5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318240    5072 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318245    5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318249    5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318253    5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318256    5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318258    5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318261    5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318263    5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318266    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318268    5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:00.320119 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318271    5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318274    5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318276    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318279    5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318281    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318284    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318286    5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318289    5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318291    5072 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318294    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318297    5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318300    5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318302    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318305    5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318308    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318310    5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318313    5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318315    5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318318    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318320    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:00.320598 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318323    5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318326    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318328    5072 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318332    5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318335    5072 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318338    5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318340    5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318343    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318345    5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318348    5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318350    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318353    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318355    5072 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318358    5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318360    5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318363    5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318365    5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318368    5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318371    5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318373    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:00.321146 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318375    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318378    5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318380    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318383    5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318386    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318389    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318391    5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318394    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318396    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318399    5072 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318401    5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318404    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318406    5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318408    5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318411    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318417    5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318422    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318425    5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318428    5072 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:00.321672 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318430    5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318433    5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318436    5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318439    5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318441    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318444    5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318446    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318448    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318451    5072 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318453    5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318456    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318458    5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318461    5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318464    5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318466    5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318949    5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318953    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318956    5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318959    5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:00.322163 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318962    5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318964    5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318967    5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318970    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318972    5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318975    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318977    5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318980    5072 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318983    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318986    5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318989    5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318991    5072 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318995    5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.318998    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319001    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319003    5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319006    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319008    5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319011    5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319013    5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:00.322643 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319016    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319018    5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319021    5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319023    5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319026    5072 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319028    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319031    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319033    5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319035    5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319038    5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319040    5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319043    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319046    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319049    5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319051    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319054    5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319056    5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319059    5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319061    5072 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:00.323161 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319064    5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319066    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319069    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319087    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319090    5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319097    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319100    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319102    5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319105    5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319107    5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319110    5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319112    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319115    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319118    5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319121    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319123    5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319126    5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319130    5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319133    5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:00.323633 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319136    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319139    5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319141    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319144    5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319147    5072 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319149    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319152    5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319154    5072 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319157    5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319160    5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319162    5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319164    5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319167    5072 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319169    5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319172    5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319174    5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319177    5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319180    5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319183    5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319186    5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:00.324271 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319188    5072 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319191    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319193    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319195    5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319200    5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319204    5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319208    5072 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319211    5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319213    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319216    5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319218    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319221    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319223    5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319226    5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319229    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319231    5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319233    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319236    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319238    5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:00.324830 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319242    5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319244    5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319246    5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319249    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.319251    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319345    5072 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319355    5072 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319364    5072 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319371    5072 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319378    5072 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319383    5072 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319390    5072 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319394    5072 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319398    5072 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319400    5072 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319404    5072 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319407    5072 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319410    5072 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319413    5072 flags.go:64] FLAG: --cgroup-root=""
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319417    5072 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319420    5072 flags.go:64] FLAG: --client-ca-file=""
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319423    5072 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319426    5072 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 03:57:00.325311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319440    5072 flags.go:64] FLAG: --cluster-domain=""
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319443    5072 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319446    5072 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319449    5072 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319453    5072 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319457    5072 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319459    5072 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319462    5072 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319465    5072 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319469    5072 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319472    5072 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319475    5072 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319478    5072 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319481    5072 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319485    5072 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319488    5072 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319491    5072 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319493    5072 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319496    5072 flags.go:64] FLAG: --enable-server="true"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319499    5072 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319512    5072 flags.go:64] FLAG: --event-burst="100"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319516    5072 flags.go:64] FLAG: --event-qps="50"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319519    5072 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319521    5072 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 03:57:00.325880 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319524    5072 flags.go:64] FLAG: --eviction-hard=""
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319528    5072 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319531    5072 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319534    5072 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319539    5072 flags.go:64] FLAG: --eviction-soft=""
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319542    5072 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319546    5072 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319549    5072 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319552    5072 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319554    5072 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319557    5072 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319561    5072 flags.go:64] FLAG: --feature-gates=""
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319565    5072 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319568    5072 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319571    5072 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319575    5072 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319578    5072 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319580    5072 flags.go:64] FLAG: --help="false"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319583    5072 flags.go:64] FLAG: --hostname-override="ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319586    5072 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319589    5072 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319592    5072 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319596    5072 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 03:57:00.326517 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319600    5072 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319603    5072 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319605    5072 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319608    5072 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319611    5072 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319614    5072 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319623    5072 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319626    5072 flags.go:64] FLAG: --kube-reserved=""
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319629    5072 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319632    5072 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319635    5072 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319637    5072 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319640    5072 flags.go:64] FLAG: --lock-file=""
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319644    5072 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319647    5072 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319650    5072 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319655    5072 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319658    5072 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319661    5072 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319664    5072 flags.go:64] FLAG: --logging-format="text"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319667    5072 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319670    5072 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319673    5072 flags.go:64] FLAG: --manifest-url=""
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319676    5072 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319680    5072 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 03:57:00.327333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319684    5072 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319687    5072 flags.go:64] FLAG: --max-pods="110"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319690    5072 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319693    5072 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319696    5072 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319699    5072 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319702    5072 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319705    5072 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319708    5072 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319718    5072 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319721    5072 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319724    5072 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319727    5072 flags.go:64] FLAG: --pod-cidr=""
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319730    5072 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319738    5072 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319741    5072 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319744    5072 flags.go:64] FLAG: --port="10250"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319747    5072 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319750    5072 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08d9981b5d6272cfc"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319753    5072 flags.go:64] FLAG: --qos-reserved=""
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319759    5072 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319762    5072 flags.go:64] FLAG: --register-node="true"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319764    5072 flags.go:64] FLAG: --register-with-taints=""
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319768    5072 flags.go:64] FLAG: --registry-burst="10"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319771    5072 flags.go:64] FLAG: --registry-qps="5"
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319774    5072 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 03:57:00.328057 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319777    5072 flags.go:64] FLAG: --reserved-memory=""
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319781    5072 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319784    5072 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319787    5072 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319790    5072 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319793    5072 flags.go:64] FLAG: --runonce="false"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319796    5072 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319799    5072 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319802    5072 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319805    5072 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319807    5072 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319811    5072 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319813    5072 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319816    5072 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319819    5072 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319822    5072 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319824    5072 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319827    5072 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319830    5072 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319833    5072 flags.go:64] FLAG: --system-cgroups=""
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319836    5072 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319846    5072 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319849    5072 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319852    5072 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319860    5072 flags.go:64] FLAG: --tls-min-version=""
Oct 30 03:57:00.328744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319863    5072 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319867    5072 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319870    5072 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319873    5072 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319876    5072 flags.go:64] FLAG: --v="2"
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319880    5072 flags.go:64] FLAG: --version="false"
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319884    5072 flags.go:64] FLAG: --vmodule=""
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319888    5072 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.319891    5072 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320055    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320059    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320062    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320064    5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320067    5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320069    5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320086    5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320089    5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320091    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320094    5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320097    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320099    5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:00.329369 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320102    5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320106    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320109    5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320111    5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320114    5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320117    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320119    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320122    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320131    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320134    5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320137    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320139    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320143    5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320164    5072 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320167    5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320170    5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320172    5072 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320175    5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320177    5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320180    5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:00.329869 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320182    5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320185    5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320187    5072 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320190    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320192    5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320195    5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320197    5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320200    5072 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320202    5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320205    5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320208    5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320210    5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320213    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320217    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320219    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320222    5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320224    5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320227    5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320230    5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:00.330389 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320232    5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320235    5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320243    5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320246    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320248    5072 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320252    5072 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320254    5072 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320257    5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320259    5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320262    5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320264    5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320267    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320269    5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320271    5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320274    5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320277    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320279    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320282    5072 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320284    5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320287    5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:00.330848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320291    5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320294    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320297    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320299    5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320302    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320304    5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320308    5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320310    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320313    5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320315    5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320318    5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320320    5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320323    5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320325    5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320327    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320335    5072 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320338    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320342    5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320344    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320347    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:00.331483 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320349    5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320352    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320354    5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320357    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320359    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320362    5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320373    5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320376    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320378    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320382    5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320387    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320389    5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320392    5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320395    5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320397    5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:00.332466 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.320424    5072 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320476    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320481    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320486    5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320489    5072 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320492    5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320494    5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320497    5072 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320499    5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320502    5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320504    5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320507    5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320510    5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320520    5072 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320523    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320526    5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320529    5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320531    5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320534    5072 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:00.332832 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320536    5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320539    5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320541    5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320544    5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320546    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320549    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320551    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320554    5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320556    5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320559    5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320561    5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320564    5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320566    5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320569    5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320571    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320573    5072 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320577    5072 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320580    5072 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320582    5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320585    5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:00.333350 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320587    5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320590    5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320592    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320595    5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320598    5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320600    5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320603    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320611    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320615    5072 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320618    5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320621    5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320624    5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320628    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320631    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320633    5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320636    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320638    5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320641    5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320643    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:00.333834 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320646    5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320648    5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320651    5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320653    5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320656    5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320658    5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320661    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320663    5072 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320665    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320669    5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320671    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320674    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320676    5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320679    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320681    5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320684    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320686    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320688    5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320691    5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:00.334335 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320693    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320696    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320707    5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320711    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320714    5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320716    5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320719    5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320721    5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320724    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320727    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320729    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320731    5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320734    5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320737    5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320739    5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320742    5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320744    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320747    5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320749    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:00.334935 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320752    5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320754    5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320757    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320761    5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320763    5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320766    5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320768    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320771    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320774    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320776    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.320779    5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.325096    5072 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.325113    5072 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.325137    5072 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.325143    5072 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325250    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325256    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325261    5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325265    5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325269    5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:00.335425 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325273    5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325278    5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325282    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325287    5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325291    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325295    5072 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325300    5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325304    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325308    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325313    5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325317    5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325321    5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325326    5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325331    5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325335    5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325340    5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325345    5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325349    5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325353    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325357    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:00.335912 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325361    5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325365    5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325369    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325373    5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325377    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325381    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325385    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325390    5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325394    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325398    5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325403    5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325407    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325411    5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325415    5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325419    5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325424    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325428    5072 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325435    5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325442    5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:00.336411 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325447    5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325452    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325457    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325462    5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325467    5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325471    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325475    5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325479    5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325484    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325488    5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325492    5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325496    5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325500    5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325505    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325509    5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325513    5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325517    5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325522    5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325525    5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:00.336862 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325529    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325533    5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325537    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325541    5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325546    5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325549    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325553    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325557    5072 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325561    5072 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325566    5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325570    5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325577    5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325582    5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325587    5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325590    5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325593    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325596    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325598    5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325601    5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325603    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:00.337328 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325606    5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325608    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325611    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325613    5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325616    5072 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325618    5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325621    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325624    5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325626    5072 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325629    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325631    5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325634    5072 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325636    5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325639    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325641    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325644    5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325646    5072 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325649    5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325651    5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325654    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:00.337792 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325656    5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325658    5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325661    5072 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.325690    5072 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325874    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325879    5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325882    5072 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325885    5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325888    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325891    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325893    5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325896    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325899    5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325901    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325903    5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325906    5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:00.338290 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325908    5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325911    5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325914    5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325916    5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325918    5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325921    5072 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325924    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325926    5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325929    5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325931    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325933    5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325936    5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325938    5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325941    5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325943    5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325945    5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325948    5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325950    5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325953    5072 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:00.338848 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325955    5072 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325958    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325965    5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325968    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325972    5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325976    5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325978    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325981    5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325984    5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325987    5072 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325989    5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325992    5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325995    5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.325997    5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326000    5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326002    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326005    5072 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326008    5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326010    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:00.339430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326012    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326015    5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326017    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326020    5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326022    5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326025    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326027    5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326029    5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326032    5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326034    5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326037    5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326039    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326042    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326044    5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326046    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326049    5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326056    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326059    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326064    5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326066    5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:00.339954 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326069    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326085    5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326089    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326092    5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326094    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326097    5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326100    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326103    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326105    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326107    5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326110    5072 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326113    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326115    5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326118    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326121    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326123    5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326126    5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326128    5072 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326130    5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:00.340524 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326133    5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326135    5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326138    5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326140    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326143    5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326145    5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326148    5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326150    5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326153    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326155    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326165    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326169    5072 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326171    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326175    5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326179    5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326183    5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:00.341036 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326186    5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.326210    5072 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326254    5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326258    5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326261    5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326264    5072 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326267    5072 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326269    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326272    5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326275    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326278    5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326282    5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326285    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326288    5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326290    5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326293    5072 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:00.341446 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326295    5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326298    5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326300    5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326302    5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326305    5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326307    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326310    5072 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326312    5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326315    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326317    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326319    5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326327    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326332    5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326334    5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326337    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326340    5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326343    5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326345    5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326348    5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:00.341842 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326351    5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326353    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326356    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326358    5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326361    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326363    5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326366    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326368    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326371    5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326373    5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326376    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326378    5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326380    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326383    5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326385    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326388    5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326390    5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326393    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326395    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326398    5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:00.342348 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326400    5072 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326403    5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326405    5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326407    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326410    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326419    5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326422    5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326424    5072 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326428    5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326430    5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326433    5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326435    5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326438    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326440    5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326443    5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326445    5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326448    5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326450    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326453    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326456    5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:00.342863 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326458    5072 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326461    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326464    5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326467    5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326470    5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326473    5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326476    5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326478    5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326481    5072 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326483    5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326486    5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326489    5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326491    5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326494    5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326496    5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326499    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326501    5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326505    5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326512    5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326515    5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:00.343453 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326519    5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326521    5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326524    5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326527    5072 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326529    5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326532    5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326535    5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326537    5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326540    5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326543    5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326546    5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326548    5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:00.326551    5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.326607    5072 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.328888    5072 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.328943    5072 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.329152    5072 server.go:1014] "Starting client certificate rotation"
Oct 30 03:57:00.343960 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.329286    5072 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 03:57:00.344689 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.329350    5072 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 03:51:18 +0000 UTC" deadline="2028-05-07 10:57:42.205503753 +0000 UTC"
Oct 30 03:57:00.344689 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.329386    5072 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13327h0m41.87611976s"
Oct 30 03:57:00.344689 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.329643    5072 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:57:00.344689 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.330715    5072 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:57:00.344689 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.331965    5072 log.go:25] "Validated CRI v1 runtime API"
Oct 30 03:57:00.344689 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.333605    5072 log.go:25] "Validated CRI v1 image API"
Oct 30 03:57:00.344689 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.334375    5072 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 03:57:00.344689 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.336305    5072 fs.go:135] Filesystem UUIDs: map[40fa5620-ebf4-4db7-af42-980fddacc43a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f8891625-04a4-4fa0-925c-b36013e04348:/dev/nvme0n1p4]
Oct 30 03:57:00.345056 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.336324    5072 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1064bba8f6bb82f69693e1e393cf229edd2cc40d911a48287ba788f2acc4a59e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1064bba8f6bb82f69693e1e393cf229edd2cc40d911a48287ba788f2acc4a59e/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18eed020933973ad437dd0e5595f20105d27486ffa38d4da81dabf0cec261f3b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18eed020933973ad437dd0e5595f20105d27486ffa38d4da81dabf0cec261f3b/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23096442889c9f8710b74a793b9bc069ff96edf5d29beb254ad357e11d50f724/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23096442889c9f8710b74a793b9bc069ff96edf5d29beb254ad357e11d50f724/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2353c7b074a790f5e9c847cbc30bb3566ed8cdb8aafa871553776ea0972f8f75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2353c7b074a790f5e9c847cbc30bb3566ed8cdb8aafa871553776ea0972f8f75/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d10c03b71d397c3dd99473e04d2d75934fdf83895543ead219e640e01653439/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d10c03b71d397c3dd99473e04d2d75934fdf83895543ead219e640e01653439/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f54ad826dc14af64a98e5eef31e80bd30ee7dd25712a14a85bad38fad43b133/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f54ad826dc14af64a98e5eef31e80bd30ee7dd25712a14a85bad38fad43b133/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f6923c74cb8d51b74345c2b62cf915662d2c876238e315d471e4308bfcc82fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f6923c74cb8d51b74345c2b62cf915662d2c876238e315d471e4308bfcc82fb/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48d9a1c03192aaa78748b3f27418c14a89cf1cd766a933260e3f9f05b6f6fcdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48d9a1c03192aaa78748b3f27418c14a89cf1cd766a933260e3f9f05b6f6fcdb/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/692e3aaa48a317e6b6d35c21fae3e7f30a86da5907eef5cb35b48d4630d3e0f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/692e3aaa48a317e6b6d35c21fae3e7f30a86da5907eef5cb35b48d4630d3e0f7/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71809b6f9c61f215314c5d4a89b2bd5059bbaf724679fa389a0239400a4e0e41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71809b6f9c61f215314c5d4a89b2bd5059bbaf724679fa389a0239400a4e0e41/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71b0828b77c7b2bc8ef6dae22ed644fe424ae384c7d887b6879527b1c4e121f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71b0828b77c7b2bc8ef6dae22ed644fe424ae384c7d887b6879527b1c4e121f5/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82fb3af131c173a4d8316136f1fe5412310db2c795ee675e292fc4951e2a4b32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82fb3af131c173a4d8316136f1fe5412310db2c795ee675e292fc4951e2a4b32/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85aded1675877ff70bc4dcb6aadcfa8a55557c0e994618f8edbe4d3a564730f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85aded1675877ff70bc4dcb6aadcfa8a55557c0e994618f8edbe4d3a564730f1/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89472633618f57002f7eef82f103ad7c8616a1475fe681f81d5a48004b399602/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89472633618f57002f7eef82f103ad7c8616a1475fe681f81d5a48004b399602/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92be8bdf88a67f238f28702f1b9f2651bc453f29c06407902f955c491d8d1d9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92be8bdf88a67f238f28702f1b9f2651bc453f29c06407902f955c491d8d1d9d/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d5d9b77363a73903c75ec607a0ee3521499392ced545186e6663f7bb9b94cc7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d5d9b77363a73903c75ec607a0ee3521499392ced545186e6663f7bb9b94cc7/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a11b2554a66ce72684088750cca0389fa8e7e1900e259271cf46e2db0aeb7acc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a11b2554a66ce72684088750cca0389fa8e7e1900e259271cf46e2db0aeb7acc/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab712e529ca78f102a7a2ab9ce589176d2173167965d7792ab6dcf8e012e4f65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab712e529ca78f102a7a2ab9ce589176d2173167965d7792ab6dcf8e012e4f65/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd7419982764252cd8990d04642082006bd671ab66a55dbf49c55df7d73e5b0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd7419982764252cd8990d04642082006bd671ab66a55dbf49c55df7d73e5b0b/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf454773ec65875e68dd75c8c21151b3f7f117b638fdbb69e91cbdc7770d4cd6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf454773ec65875e68dd75c8c21151b3f7f117b638fdbb69e91cbdc7770d4cd6/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3eb51613da26058dac2cbc5e835f79a7d4ac04cfae6ab2e2184d5017fde4878/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3eb51613da26058dac2cbc5e835f79a7d4ac04cfae6ab2e2184d5017fde4878/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cea3f6d1877d53e809bcc16713de506133dc4f5201af4ff00915e321b5a6cf17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cea3f6d1877d53e809bcc16713de506133dc4f5201af4ff00915e321b5a6cf17/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0353988f43af1839855daa5855f75647988e0f6163e7790e44cd475190d00eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0353988f43af1839855daa5855f75647988e0f6163e7790e44cd475190d00eb/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de458eaf8dd33fa9b7e09166d8cfdf13201dbdece842f3396a4b802ec4da6d8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de458eaf8dd33fa9b7e09166d8cfdf13201dbdece842f3396a4b802ec4da6d8b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3f807feb80a7f1d40ed6617d546ee2855991902856e0a2a30ee79d0325386c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3f807feb80a7f1d40ed6617d546ee2855991902856e0a2a30ee79d0325386c7/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2c832f8ad8bb903525ec645e2610c91ec3478cc3f73deca60f4b1ba28e3ec15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2c832f8ad8bb903525ec645e2610c91ec3478cc3f73deca60f4b1ba28e3ec15/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0384e218-1937-4099-8566-67775cda0521/volumes/kubernetes.io~projected/kube-api-access-lpt9x:{mountpoint:/var/lib/kubelet/pods/0384e218-1937-4099-8566-67775cda0521/volumes/kubernetes.io~projected/kube-api-access-lpt9x major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d2109e3-d762-4bf9-bf3a-d405148ec32c/volumes/kubernetes.io~projected/kube-api-access-s9hcr:{mountpoint:/var/lib/kubelet/pods/0d2109e3-d762-4bf9-bf3a-d405148ec32c/volumes/kubernetes.io~projected/kube-api-access-s9hcr major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d40d8b3-4844-4a76-b395-b5b8a2a41438/volumes/kubernetes.io~projected/kube-api-access-6tnhw:{mountpoint:/var/lib/kubelet/pods/1d40d8b3-4844-4a76-b395-b5b8a2a41438/volumes/kubernetes.io~projected/kube-api-access-6tnhw major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fe5ff28-0896-4f54-b5cc-5326a0d60752/volumes/kubernetes.io~projected/kube-api-access-twx6b:{mountpoint:/var/lib/kubelet/pods/1fe5ff28-0896-4f54-b5cc-5326a0d60752/volumes/kubernetes.io~projected/kube-api-access-twx6b major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fe5ff28-0896-4f54-b5cc-5326a0d60752/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1fe5ff28-0896-4f54-b5cc-5326a0d60752/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes/kubernetes.io~projected/kube-api-access-vzdlq:{mountpoint:/var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes/kubernetes.io~projected/kube-api-access-vzdlq major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes/kubernetes.io~projected/kube-api-access-99bzl:{mountpoint:/var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes/kubernetes.io~projected/kube-api-access-99bzl major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41551065-0510-4289-a742-83987e916774/volumes/kubernetes.io~projected/kube-api-access-4ntgd:{mountpoint:/var/lib/kubelet/pods/41551065-0510-4289-a742-83987e916774/volumes/kubernetes.io~projected/kube-api-access-4ntgd major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4455e2df-ec71-4748-8920-36b2374af2a1/volumes/kubernetes.io~projected/kube-api-access-gxmnr:{mountpoint:/var/lib/kubelet/pods/4455e2df-ec71-4748-8920-36b2374af2a1/volumes/kubernetes.io~projected/kube-api-access-gxmnr major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/478b6cf0-d0dd-4d65-aafb-ff211b60c188/volumes/kubernetes.io~projected/kube-api-access-295lb:{mountpoint:/var/lib/kubelet/pods/478b6cf0-d0dd-4d65-aafb-ff211b60c188/volumes/kubernetes.io~projected/kube-api-access-295lb major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/491c21e4-c0e9-4d60-a00e-c896bcda29f7/volumes/kubernetes.io~projected/kube-api-access-m2qhn:{mountpoint:/var/lib/kubelet/pods/491c21e4-c0e9-4d60-a00e-c896bcda29f7/volumes/kubernetes.io~projected/kube-api-access-m2qhn major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c692538-d934-46cf-8c4c-d6a888dd00a1/volumes/kubernetes.io~projected/kube-api-access-cr8xt:{mountpoint:/var/lib/kubelet/pods/4c692538-d934-46cf-8c4c-d6a888dd00a1/volumes/kubernetes.io~projected/kube-api-access-cr8xt major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b/volumes/kubernetes.io~projected/kube-api-access-28tfx:{mountpoint:/var/lib/kubelet/pods/4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b/volumes/kubernetes.io~projected/kube-api-access-28tfx major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/503b82e1-e56c-4c80-9d63-fa8e647c0991/volumes/kubernetes.io~projected/kube-api-access-27j9s:{mountpoint:/var/lib/kubelet/pods/503b82e1-e56c-4c80-9d63-fa8e647c0991/volumes/kubernetes.io~projected/kube-api-access-27j9s major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a290674-68db-4847-b2a1-ad968844f699/volumes/kubernetes.io~projected/kube-api-access-6jq7h:{mountpoint:/var/lib/kubelet/pods/5a290674-68db-4847-b2a1-ad968844f699/volumes/kubernetes.io~projected/kube-api-access-6jq7h major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e68d10c-d217-40d3-945d-c87ccbd7434b/volumes/kubernetes.io~projected/kube-api-access-9fnjr:{mountpoint:/var/lib/kubelet/pods/5e68d10c-d217-40d3-945d-c87ccbd7434b/volumes/kubernetes.io~projected/kube-api-access-9fnjr major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/623e4b93-06e4-44f9-bd85-d9b663df4834/volumes/kubernetes.io~projected/kube-api-access-6t2hj:{mountpoint:/var/lib/kubelet/pods/623e4b93-06e4-44f9-bd85-d9b663df4834/volumes/kubernetes.io~projected/kube-api-access-6t2hj major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67dad9d6-9671-49f3-84a2-024463451062/volumes/kubernetes.io~projected/kube-api-access-xq4h9:{mountpoint:/var/lib/kubelet/pods/67dad9d6-9671-49f3-84a2-024463451062/volumes/kubernetes.io~projected/kube-api-access-xq4h9 major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221/volumes/kubernetes.io~projected/kube-api-access-cktnq:{mountpoint:/var/lib/kubelet/pods/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221/volumes/kubernetes.io~projected/kube-api-access-cktnq major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/730fcc37-b18e-46e3-aea5-a7fba60b7a8c/volumes/kubernetes.io~projected/kube-api-access-7q52f:{mountpoint:/var/lib/kubelet/pods/730fcc37-b18e-46e3-aea5-a7fba60b7a8c/volumes/kubernetes.io~projected/kube-api-access-7q52f major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/788f5892-601d-42cc-b597-528a091feac9/volumes/kubernetes.io~projected/kube-api-access-n9z28:{mountpoint:/var/lib/kubelet/pods/788f5892-601d-42cc-b597-528a091feac9/volumes/kubernetes.io~projected/kube-api-access-n9z28 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67/volumes/kubernetes.io~projected/kube-api-access-6mlbm:{mountpoint:/var/lib/kubelet/pods/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67/volumes/kubernetes.io~projected/kube-api-access-6mlbm major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67/volumes/kubernetes.io~secret/signing-key major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89f2993a-ed5d-488e-832d-bcba8ddd9706/volumes/kubernetes.io~projected/kube-api-access-9b9tt:{mountpoint:/var/lib/kubelet/pods/89f2993a-ed5d-488e-832d-bcba8ddd9706/volumes/kubernetes.io~projected/kube-api-access-9b9tt major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89f2993a-ed5d-488e-832d-bcba8ddd9706/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/89f2993a-ed5d-488e-832d-bcba8ddd9706/volumes/kubernetes.io~secret/cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e7b0fed-186b-4d7b-affa-e91d5ef4b523/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8e7b0fed-186b-4d7b-affa-e91d5ef4b523/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/907912b8-9313-43b3-b324-e8498ea6519a/volumes/kubernetes.io~projected/kube-api-access-5mq42:{mountpoint:/var/lib/kubelet/pods/907912b8-9313-43b3-b324-e8498ea6519a/volumes/kubernetes.io~projected/kube-api-access-5mq42 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/907912b8-9313-43b3-b324-e8498ea6519a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/907912b8-9313-43b3-b324-e8498ea6519a/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99e3f70c-6dd8-467e-a444-7692e30a76a4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/99e3f70c-6dd8-467e-a444-7692e30a76a4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99e3f70c-6dd8-467e-a444-7692e30a76a4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/99e3f70c-6dd8-467e-a444-7692e30a76a4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99e3f70c-6dd8-467e-a444-7692e30a76a4/volumes/kubernetes.io~projected/kube-api-access-78q6r:{mountpoint:/var/lib/kubelet/pods/99e3f70c-6dd8-467e-a444-7692e30a76a4/volumes/kubernetes.io~projected/kube-api-access-78q6r major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e6da480-8072-42d2-98f3-91566744aa96/volumes/kubernetes.io~projected/kube-api-access-5mr28:{mountpoint:/var/lib/kubelet/pods/9e6da480-8072-42d2-98f3-91566744aa96/volumes/kubernetes.io~projected/kube-api-access-5mr28 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b555cf82-5423-4c47-a004-ae399654683b/volumes/kubernetes.io~projected/kube-api-access-5nfhp:{mountpoint:/var/lib/kubelet/pods/b555cf82-5423-4c47-a004-ae399654683b/volumes/kubernetes.io~projected/kube-api-access-5nfhp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b752c121-5e44-4df2-9f17-97f0c899516b/volumes/kubernetes.io~projected/kube-api-access-n278c:{mountpoint:/var/lib/kubelet/pods/b752c121-5e44-4df2-9f17-97f0c899516b/volumes/kubernetes.io~projected/kube-api-access-n278c major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b752c121-5e44-4df2-9f17-97f0c899516b/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/b752c121-5e44-4df2-9f17-97f0c899516b/volumes/kubernetes.io~secret/default-certificate major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b752c121-5e44-4df2-9f17-97f0c899516b/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/b752c121-5e44-4df2-9f17-97f0c899516b/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc9a426a-587d-4611-9298-90fc4b4f1dc3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bc9a426a-587d-4611-9298-90fc4b4f1dc3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6fec406-0515-44b5-ae95-c2fbfe84b452/volumes/kubernetes.io~projected/kube-api-access-qcl9n:{mountpoint:/var/lib/kubelet/pods/c6fec406-0515-44b5-ae95-c2fbfe84b452/volumes/kubernetes.io~projected/kube-api-access-qcl9n major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbf6ac89-eb35-4d19-a28c-6f879f02703a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/cbf6ac89-eb35-4d19-a28c-6f879f02703a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbf6ac89-eb35-4d19-a28c-6f879f02703a/volumes/kubernetes.io~projected/kube-api-access-h4n7t:{mountpoint:/var/lib/kubelet/pods/cbf6ac89-eb35-4d19-a28c-6f879f02703a/volumes/kubernetes.io~projected/kube-api-access-h4n7t major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbf6ac89-eb35-4d19-a28c-6f879f02703a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/cbf6ac89-eb35-4d19-a28c-6f879f02703a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d18cc860-7cd8-48d9-9a7a-40a40a53f379/volumes/kubernetes.io~projected/kube-api-access-ppk4b:{mountpoint:/var/lib/kubelet/pods/d18cc860-7cd8-48d9-9a7a-40a40a53f379/volumes/kubernetes.io~projected/kube-api-access-ppk4b major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d41c0686-14d0-43c4-928e-e38b57021fb7/volumes/kubernetes.io~projected/kube-api-access-bp4k9:{mountpoint:/var/lib/kubelet/pods/d41c0686-14d0-43c4-928e-e38b57021fb7/volumes/kubernetes.io~projected/kube-api-access-bp4k9 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d41c0686-14d0-43c4-928e-e38b57021fb7/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d41c0686-14d0-43c4-928e-e38b57021fb7/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/8e8a4ddbb8666e35fc57995b3a3eff55c0688d51ff64bf73c3bd4a378bb57097/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/100c8c470b3af77a9e7279ba3fd00c8db2df471987d4f10643dba4717acf06b6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3a2c7387396006bd9201746a763c2643a63b2672f78f5f30472d9a6aeeef5274/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/12279a10aa5883dd2a945cc56f8ee2aa316b37a694c418a8d758400619b149e3/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/df34cce20415b6e898b16141d787ded2061a73a3666cf2105e44f4de2dc4fbd9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/26bccd448fd165dbf8a65a73e3a6e24511ffaa7b0242d9fbcc3f247abd14c1dd/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/98ddaf87335b94b281e91a5b4ee489a6ba16d5c3c27832e94cb1f2ffbe8b0991/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2ff913be1288e094e4c68003ca51d3cec2e0e68272f0dda6cdb3218f314ea610/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/b051361b91d870e8110b27a0d332aea932181fd86ae5c7dc1bb30038a8950229/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/71f7938834dca9476d86aec70aec5d5e3450aebd2a6aeb0db4d48190137a2e1d/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/f6305898e4b3fd4a7b56e4a5082ecdaf5b256492400b2c5076be59e5b851eba1/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/c09cc0e3679ab43257c7cff53b4b41a943cb24831f66d10d7d488641eac6a72d/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/939fc441432552492abf099e1f7e70ff0da2d09d70e0ecfa2020baa61d41e5cf/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ce67fb22809a6dee8c98e8aa0f14a650d5583a4042a7b91880471901455e4c51/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1c65233fe3559c4b085517b5780af9378afc433757d1ad319cd230412be0e936/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/ed9bd9d5f1e6c59da54de5193edb120ef5d0cc18f4c198ff5be3c7388e361dfc/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2897dbae15a48ba8a695763046d0e3c38d90c0b62e658ef2d5dd7d35a14ac7e7/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/90fa17303c904a1bef2921ed14cdb623d00c3dbbe887bbc202feda0735a33862/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/01401f2b354da8c015b73586229c24a663801e9c2e54c79aa6d3927c2cc18bc3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a1b3197ed7511643b6dcbc63eb78d319a4c8e9314b3366266834f06bad84c6b0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/772a50adaad20613d6f84baebc4d7339a798f63f75e93f6f0a0032c55507e6bc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/9e223b5861bd5a6617f2a49ae0ce59f928fa5457db7b7996c6dbb6c09eb175b7/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/d8e53a887e4082e416e3816306b5bc59c8a1bb90f3264bac07c0e94701dd7188/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/cf77c961bf92cde69db1dd5fda4eadab0f4cd2e42d84382ebf47805934562f77/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/81b013407eb8b826af30450f4de9ca7dc2e05712d2544b15de8a106449653a25/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/1c462d8c0d4a58ecab7a71698c5d0d254ce070b13e28d859f9b18439d9040d0a/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/369dd8eb79a7d5e928d9a6137b1a884e58f0487d35498cd3eee360e4de118bb5/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/76a9de6e9f3f14d01d5a392247d36dcee8ad3a0e521f6aa9f28dd2bc89db8937/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/598853c0f2138ae09cb0fc77e2573b4dd0a62c65f808889c8ef382be2bec751d/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/e504ed6fcfb82715086af12e7436ff1d7f1e9964cf9729d8643d9a40fddfb1af/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/3c978023a4f4c18dfba4497c8cdd3d72c9c5d31b3c2859ceff82e395a2076b2a/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/a7fdb2edb81741b4e9f271f5bedd6d9283379ca032bacbed9d3df74558ecb179/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/3b00dba1d23a5a23d0138c1ab22dc6dad5cf79f8ab62b248b46f0fbd6f90effb/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/0c5c6188db1c020709e9a111fb560553d5c702cfc212ec10373fc330a4dc44c2/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/4b4612bbd78aafcff1d3f3bb4542840b24715fb13092b933fefa123fe7236511/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/a775557e27fc6c8f75e89b66de51372003610385cf33fd84760d6fbd28fc5702/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/0f09ba5dd56e48a2db2294f33fa2ab263e11fe41d87675a08d3cdc4f31736999/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/24150a3da0138bdf0e888be2f17fb4bb7b4d212a01ef097f93ba30a4ff174c82/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/643797a7cef4a57c4ab97c0ea1d37dd347f791dc8c9afbdc04e140da63f49ed1/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/78ca5ab07a6b76cfacc0a1bc70d48d6cacd37699f23b06fd6862553c01ec84e1/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/119d5527798c38d032b8b81082dba5c718ef9b856e2e12fd6abd10e8ff57ae6b/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/810773855e82f32bda60d50220826d1751336949c54d68201fcf6077dd4b0d7b/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/05efee7e7e3d8acead81acab2b2d0d88d912b74925f5c9326b98561027b008c9/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/28f79b07109956a7e09e9545fead2a00c08f7061fd122749cc336b29921d392d/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/1752428f90380800ece055a2ffc57261770ebe5fdf3531484631306a5f938abf/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/c5faf413671071ce4d305cc83f3626fb96e1d4a5f363cb11461b20e65d3e8144/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/14d1934bf1dedb78325c1314a8ff60b2e0ccd5ac29fda93f41436f311f84064e/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/dc886c50df392311fe9e7644c79a20c0962b4bf0b1e8f5de6c4092f8f44170ab/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/39805dd402a109e77524b1ffca85b989c8801bf620137a7eac8547f4f1ca906c/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/779a17c1d0697f6fcc5680e0b8a1f730890486b47b7d26a1175ed10a7afd13c8/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/f15167d6a4e86b405f2628417a09ba60168a9062b73e5a0b8eb8a3fe22ecbabf/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/fc055c123ed8a92afef27d78cbd0c05d2419651012d0e80df4f758cb775c039f/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/251836fe7c389c6573e78740b85b189d20470c4dab722e4e7c575334805c7b31/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/86b18c3661b424b55362e979d48b3809b08acd67d3258efd13494d656a2120bf/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/31be6f84d1ca25aee715671dc5826fef17d16d91623a4751d86fe68b2989e0a4/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/e3dc6de7b2b95b9b2a63b2b8924119e476a74b65c42b59d7b1d239a1823f0adf/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/b86adc6633c4af64ca8c4c4fae72d88aefc2e597961758364f9cda0d3041d808/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/d8cacc160ab01d08d7205e435175bdd40dddef3e5227e9dcae7a4d5bb8c91ed6/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/5d2cbcabf895403401fb40ddd3693b77c31b36f91ed67e4337d09269083d2295/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/84d51dfba427d874b7781a37cb7ecd68beb1a95628d59d5f4fbf51cc5813801a/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/f2ef0ae59c0940360c45e92ab3819953004f8c491219749136ae86065dc3709d/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/3e47bce8384022edb5e864fbd56d3a422a289f5f8a7c1e6f94336ac494946016/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 03:57:00.346355 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.345990    5072 manager.go:217] Machine: {Timestamp:2026-10-30 03:57:00.345368281 +0000 UTC m=+0.086791370 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102774 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21eca4969f6e2b5bed4d0fcc570234 SystemUUID:ec21eca4-969f-6e2b-5bed-4d0fcc570234 BootID:fd54cb52-50c0-4f5b-b1d8-116a83ab07ae Filesystems:[{Device:/run/containers/storage/overlay-containers/e3f807feb80a7f1d40ed6617d546ee2855991902856e0a2a30ee79d0325386c7/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/99e3f70c-6dd8-467e-a444-7692e30a76a4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbf6ac89-eb35-4d19-a28c-6f879f02703a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cea3f6d1877d53e809bcc16713de506133dc4f5201af4ff00915e321b5a6cf17/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c692538-d934-46cf-8c4c-d6a888dd00a1/volumes/kubernetes.io~projected/kube-api-access-cr8xt DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d41c0686-14d0-43c4-928e-e38b57021fb7/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4455e2df-ec71-4748-8920-36b2374af2a1/volumes/kubernetes.io~projected/kube-api-access-gxmnr DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:350 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de458eaf8dd33fa9b7e09166d8cfdf13201dbdece842f3396a4b802ec4da6d8b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f6923c74cb8d51b74345c2b62cf915662d2c876238e315d471e4308bfcc82fb/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d5d9b77363a73903c75ec607a0ee3521499392ced545186e6663f7bb9b94cc7/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1064bba8f6bb82f69693e1e393cf229edd2cc40d911a48287ba788f2acc4a59e/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89472633618f57002f7eef82f103ad7c8616a1475fe681f81d5a48004b399602/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbf6ac89-eb35-4d19-a28c-6f879f02703a/volumes/kubernetes.io~projected/kube-api-access-h4n7t DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d40d8b3-4844-4a76-b395-b5b8a2a41438/volumes/kubernetes.io~projected/kube-api-access-6tnhw DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/692e3aaa48a317e6b6d35c21fae3e7f30a86da5907eef5cb35b48d4630d3e0f7/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf454773ec65875e68dd75c8c21151b3f7f117b638fdbb69e91cbdc7770d4cd6/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71809b6f9c61f215314c5d4a89b2bd5059bbaf724679fa389a0239400a4e0e41/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221/volumes/kubernetes.io~projected/kube-api-access-cktnq DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3eb51613da26058dac2cbc5e835f79a7d4ac04cfae6ab2e2184d5017fde4878/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92be8bdf88a67f238f28702f1b9f2651bc453f29c06407902f955c491d8d1d9d/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/491c21e4-c0e9-4d60-a00e-c896bcda29f7/volumes/kubernetes.io~projected/kube-api-access-m2qhn DeviceMajor:0 DeviceMinor:340 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e68d10c-d217-40d3-945d-c87ccbd7434b/volumes/kubernetes.io~projected/kube-api-access-9fnjr DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d2109e3-d762-4bf9-bf3a-d405148ec32c/volumes/kubernetes.io~projected/kube-api-access-s9hcr DeviceMajor:0 DeviceMinor:316 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbf6ac89-eb35-4d19-a28c-6f879f02703a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85aded1675877ff70bc4dcb6aadcfa8a55557c0e994618f8edbe4d3a564730f1/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/41551065-0510-4289-a742-83987e916774/volumes/kubernetes.io~projected/kube-api-access-4ntgd DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab712e529ca78f102a7a2ab9ce589176d2173167965d7792ab6dcf8e012e4f65/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f54ad826dc14af64a98e5eef31e80bd30ee7dd25712a14a85bad38fad43b133/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23096442889c9f8710b74a793b9bc069ff96edf5d29beb254ad357e11d50f724/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e6da480-8072-42d2-98f3-91566744aa96/volumes/kubernetes.io~projected/kube-api-access-5mr28 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89f2993a-ed5d-488e-832d-bcba8ddd9706/volumes/kubernetes.io~projected/kube-api-access-9b9tt DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b752c121-5e44-4df2-9f17-97f0c899516b/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b752c121-5e44-4df2-9f17-97f0c899516b/volumes/kubernetes.io~projected/kube-api-access-n278c DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1fe5ff28-0896-4f54-b5cc-5326a0d60752/volumes/kubernetes.io~projected/kube-api-access-twx6b DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes/kubernetes.io~projected/kube-api-access-vzdlq DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/99e3f70c-6dd8-467e-a444-7692e30a76a4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0384e218-1937-4099-8566-67775cda0521/volumes/kubernetes.io~projected/kube-api-access-lpt9x DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48d9a1c03192aaa78748b3f27418c14a89cf1cd766a933260e3f9f05b6f6fcdb/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71b0828b77c7b2bc8ef6dae22ed644fe424ae384c7d887b6879527b1c4e121f5/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18eed020933973ad437dd0e5595f20105d27486ffa38d4da81dabf0cec261f3b/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/907912b8-9313-43b3-b324-e8498ea6519a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/907912b8-9313-43b3-b324-e8498ea6519a/volumes/kubernetes.io~projected/kube-api-access-5mq42 DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d18cc860-7cd8-48d9-9a7a-40a40a53f379/volumes/kubernetes.io~projected/kube-api-access-ppk4b DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/478b6cf0-d0dd-4d65-aafb-ff211b60c188/volumes/kubernetes.io~projected/kube-api-access-295lb DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/623e4b93-06e4-44f9-bd85-d9b663df4834/volumes/kubernetes.io~projected/kube-api-access-6t2hj DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/503b82e1-e56c-4c80-9d63-fa8e647c0991/volumes/kubernetes.io~projected/kube-api-access-27j9s DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/67dad9d6-9671-49f3-84a2-024463451062/volumes/kubernetes.io~projected/kube-api-access-xq4h9 DeviceMajor:0 DeviceMinor:317 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0353988f43af1839855daa5855f75647988e0f6163e7790e44cd475190d00eb/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99e3f70c-6dd8-467e-a444-7692e30a76a4/volumes/kubernetes.io~projected/kube-api-access-78q6r DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d41c0686-14d0-43c4-928e-e38b57021fb7/volumes/kubernetes.io~projected/kube-api-access-bp4k9 DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a11b2554a66ce72684088750cca0389fa8e7e1900e259271cf46e2db0aeb7acc/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d10c03b71d397c3dd99473e04d2d75934fdf83895543ead219e640e01653439/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82fb3af131c173a4d8316136f1fe5412310db2c795ee675e292fc4951e2a4b32/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bc9a426a-587d-4611-9298-90fc4b4f1dc3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67/volumes/kubernetes.io~projected/kube-api-access-6mlbm DeviceMajor:0 DeviceMinor:353 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/1fe5ff28-0896-4f54-b5cc-5326a0d60752/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b/volumes/kubernetes.io~projected/kube-api-access-28tfx DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/788f5892-601d-42cc-b597-528a091feac9/volumes/kubernetes.io~projected/kube-api-access-n9z28 DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes/kubernetes.io~projected/kube-api-access-99bzl DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2c832f8ad8bb903525ec645e2610c91ec3478cc3f73deca60f4b1ba28e3ec15/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6fec406-0515-44b5-ae95-c2fbfe84b452/volumes/kubernetes.io~projected/kube-api-access-qcl9n DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/730fcc37-b18e-46e3-aea5-a7fba60b7a8c/volumes/kubernetes.io~projected/kube-api-access-7q52f DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b555cf82-5423-4c47-a004-ae399654683b/volumes/kubernetes.io~projected/kube-api-access-5nfhp DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b752c121-5e44-4df2-9f17-97f0c899516b/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/8e7b0fed-186b-4d7b-affa-e91d5ef4b523/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a290674-68db-4847-b2a1-ad968844f699/volumes/kubernetes.io~projected/kube-api-access-6jq7h DeviceMajor:0 DeviceMinor:345 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2353c7b074a790f5e9c847cbc30bb3566ed8cdb8aafa871553776ea0972f8f75/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89f2993a-ed5d-488e-832d-bcba8ddd9706/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd7419982764252cd8990d04642082006bd671ab66a55dbf49c55df7d73e5b0b/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1064bba8f6bb82f MacAddress:36:cf:67:d5:ae:08 Speed:10000 Mtu:8901} {Name:23096442889c9f8 MacAddress:92:79:b5:ac:40:1d Speed:10000 Mtu:8901} {Name:3d10c03b71d397c MacAddress:16:6e:eb:68:77:25 Speed:10000 Mtu:8901} {Name:3f6923c74cb8d51 MacAddress:3a:f9:be:f5:6f:18 Speed:10000 Mtu:8901} {Name:48d9a1c03192aaa MacAddress:5e:5e:fc:e0:47:15 Speed:10000 Mtu:8901} {Name:692e3aaa48a317e MacAddress:9e:d5:25:9a:7b:2e Speed:10000 Mtu:8901} {Name:82fb3af131c173a MacAddress:9a:73:0f:a1:68:a3 Speed:10000 Mtu:8901} {Name:92be8bdf88a67f2 MacAddress:2a:13:8f:c7:ed:e1 Speed:10000 Mtu:8901} {Name:9d5d9b77363a739 MacAddress:fe:ff:38:ef:85:48 Speed:10000 Mtu:8901} {Name:a11b2554a66ce72 MacAddress:fa:10:66:f0:03:68 Speed:10000 Mtu:8901} {Name:bd7419982764252 MacAddress:8e:4f:b6:f6:d5:79 Speed:10000 Mtu:8901} {Name:bf454773ec65875 MacAddress:02:e3:28:82:d3:8b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d4:91:05:c5 Speed:0 Mtu:9001} {Name:br-int MacAddress:fe:07:1e:3e:6f:89 Speed:0 Mtu:8901} {Name:c3eb51613da2605 MacAddress:8a:a8:bb:d9:b5:f7 Speed:10000 Mtu:8901} {Name:cea3f6d1877d53e MacAddress:f6:bd:72:44:69:f3 Speed:10000 Mtu:8901} {Name:d0353988f43af18 MacAddress:ea:e0:b1:50:06:5e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d4:91:05:c5 Speed:0 Mtu:9001} {Name:f2c832f8ad8bb90 MacAddress:b6:dc:32:a8:b4:04 Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5a:d0:ef:4b:af:25 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 03:57:00.346581 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.346360    5072 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 03:57:00.346581 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.346401    5072 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 03:57:00.346692 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.346665    5072 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 03:57:00.346841 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.346693    5072 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 03:57:00.346881 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.346844    5072 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 03:57:00.346881 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.346852    5072 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 03:57:00.346881 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.346863    5072 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 03:57:00.346881 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.346875    5072 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 03:57:00.346981 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.346893    5072 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 03:57:00.347048 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.347039    5072 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 03:57:00.347178 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.347169    5072 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 03:57:00.347218 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.347208    5072 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 03:57:00.347246 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.347226    5072 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 03:57:00.347246 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.347236    5072 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 03:57:00.347246 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.347245    5072 kubelet.go:405] "Adding apiserver pod source"
Oct 30 03:57:00.347357 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.347252    5072 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 03:57:00.348213 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.348196    5072 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 03:57:00.348213 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.348212    5072 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 03:57:00.348666 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.348651    5072 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 03:57:00.351750 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.351736    5072 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 03:57:00.351750 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.351750    5072 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 03:57:00.351856 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.351847    5072 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 03:57:00.352041 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352030    5072 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 03:57:00.352069 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352059    5072 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 03:57:00.352185 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352175    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 03:57:00.352216 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352191    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 03:57:00.352216 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352197    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 03:57:00.352216 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352202    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 03:57:00.352216 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352210    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 03:57:00.352319 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352219    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 03:57:00.352319 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352228    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 03:57:00.352319 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352233    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 03:57:00.352319 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352239    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 03:57:00.352319 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352245    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 03:57:00.352319 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352259    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 03:57:00.352319 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352269    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 03:57:00.352319 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352295    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 03:57:00.352319 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.352305    5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 03:57:00.355682 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.355669    5072 server.go:1263] "Started kubelet"
Oct 30 03:57:00.355752 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.355726    5072 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 03:57:00.356271 ip-10-0-139-99 systemd[1]: Started Kubernetes Kubelet.
Oct 30 03:57:00.356455 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.356246    5072 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 03:57:00.356455 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.356306    5072 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 03:57:00.356573 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.356556    5072 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 03:57:00.357360 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.357346    5072 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 03:57:00.362307 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.362271    5072 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 03:57:00.362632 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.362618    5072 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 03:57:00.362724 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.362634    5072 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 03:57:00.362724 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.362707    5072 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 03:57:00.362982 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:00.362962    5072 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Oct 30 03:57:00.363382 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.363368    5072 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 03:57:00.363710 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.363670    5072 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 03:51:20 +0000 UTC" deadline="2028-07-04 10:38:42.45404071 +0000 UTC"
Oct 30 03:57:00.363710 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.363711    5072 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14718h41m42.090332277s"
Oct 30 03:57:00.366597 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.366574    5072 factory.go:55] Registering systemd factory
Oct 30 03:57:00.366597 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.366599    5072 factory.go:223] Registration of the systemd container factory successfully
Oct 30 03:57:00.367406 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367369    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d41c0686-14d0-43c4-928e-e38b57021fb7" volumeName="kubernetes.io/empty-dir/d41c0686-14d0-43c4-928e-e38b57021fb7-snapshots" seLinuxMountContext=""
Oct 30 03:57:00.367489 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367409    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d41c0686-14d0-43c4-928e-e38b57021fb7" volumeName="kubernetes.io/secret/d41c0686-14d0-43c4-928e-e38b57021fb7-serving-cert" seLinuxMountContext=""
Oct 30 03:57:00.367489 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367426    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="907912b8-9313-43b3-b324-e8498ea6519a" volumeName="kubernetes.io/configmap/907912b8-9313-43b3-b324-e8498ea6519a-config" seLinuxMountContext=""
Oct 30 03:57:00.367489 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367439    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0384e218-1937-4099-8566-67775cda0521" volumeName="kubernetes.io/projected/0384e218-1937-4099-8566-67775cda0521-kube-api-access-lpt9x" seLinuxMountContext=""
Oct 30 03:57:00.367489 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367452    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="478b6cf0-d0dd-4d65-aafb-ff211b60c188" volumeName="kubernetes.io/projected/478b6cf0-d0dd-4d65-aafb-ff211b60c188-kube-api-access-295lb" seLinuxMountContext=""
Oct 30 03:57:00.367489 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367464    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a290674-68db-4847-b2a1-ad968844f699" volumeName="kubernetes.io/projected/5a290674-68db-4847-b2a1-ad968844f699-kube-api-access-6jq7h" seLinuxMountContext=""
Oct 30 03:57:00.367489 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367476    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67dad9d6-9671-49f3-84a2-024463451062" volumeName="kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367492    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d25c4cc-371f-4a3d-8f68-b3d8923f0c67" volumeName="kubernetes.io/secret/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-signing-key" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367506    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e6da480-8072-42d2-98f3-91566744aa96" volumeName="kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367521    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf6ac89-eb35-4d19-a28c-6f879f02703a" volumeName="kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovnkube-config" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367535    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="907912b8-9313-43b3-b324-e8498ea6519a" volumeName="kubernetes.io/configmap/907912b8-9313-43b3-b324-e8498ea6519a-trusted-ca" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367549    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01bc8bd9-f593-4acf-97c0-fc6c4beb4867" volumeName="kubernetes.io/configmap/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-nginx-conf" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367562    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="343419d3-210d-4c12-86ff-c92f8f574b27" volumeName="kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-trusted-ca" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367575    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d41c0686-14d0-43c4-928e-e38b57021fb7" volumeName="kubernetes.io/configmap/d41c0686-14d0-43c4-928e-e38b57021fb7-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367592    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d40d8b3-4844-4a76-b395-b5b8a2a41438" volumeName="kubernetes.io/projected/1d40d8b3-4844-4a76-b395-b5b8a2a41438-kube-api-access-6tnhw" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367606    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="343419d3-210d-4c12-86ff-c92f8f574b27" volumeName="kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367628    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4455e2df-ec71-4748-8920-36b2374af2a1" volumeName="kubernetes.io/projected/4455e2df-ec71-4748-8920-36b2374af2a1-kube-api-access-gxmnr" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367641    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="491c21e4-c0e9-4d60-a00e-c896bcda29f7" volumeName="kubernetes.io/projected/491c21e4-c0e9-4d60-a00e-c896bcda29f7-kube-api-access-m2qhn" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367655    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72c9fb60-7d7b-46b7-8d02-0b51dd1d0221" volumeName="kubernetes.io/projected/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-kube-api-access-cktnq" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367670    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99e3f70c-6dd8-467e-a444-7692e30a76a4" volumeName="kubernetes.io/projected/99e3f70c-6dd8-467e-a444-7692e30a76a4-kube-api-access-78q6r" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367682    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6fec406-0515-44b5-ae95-c2fbfe84b452" volumeName="kubernetes.io/projected/c6fec406-0515-44b5-ae95-c2fbfe84b452-kube-api-access-qcl9n" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367694    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf6ac89-eb35-4d19-a28c-6f879f02703a" volumeName="kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367706    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7b0fed-186b-4d7b-affa-e91d5ef4b523" volumeName="kubernetes.io/configmap/8e7b0fed-186b-4d7b-affa-e91d5ef4b523-konnectivity-ca" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367720    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="343419d3-210d-4c12-86ff-c92f8f574b27" volumeName="kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-kube-api-access-vzdlq" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367734    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72c9fb60-7d7b-46b7-8d02-0b51dd1d0221" volumeName="kubernetes.io/configmap/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-config" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367745    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e6da480-8072-42d2-98f3-91566744aa96" volumeName="kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-cni-binary-copy" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367759    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b752c121-5e44-4df2-9f17-97f0c899516b" volumeName="kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-stats-auth" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367771    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf6ac89-eb35-4d19-a28c-6f879f02703a" volumeName="kubernetes.io/secret/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 03:57:00.367785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367789    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="343419d3-210d-4c12-86ff-c92f8f574b27" volumeName="kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-bound-sa-token" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367810    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395d5cbd-c077-4578-b377-3034e42a2b94" volumeName="kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367822    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67dad9d6-9671-49f3-84a2-024463451062" volumeName="kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-service-ca-bundle" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367837    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0384e218-1937-4099-8566-67775cda0521" volumeName="kubernetes.io/configmap/0384e218-1937-4099-8566-67775cda0521-serviceca" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367859    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="503b82e1-e56c-4c80-9d63-fa8e647c0991" volumeName="kubernetes.io/projected/503b82e1-e56c-4c80-9d63-fa8e647c0991-kube-api-access-27j9s" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367870    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="788f5892-601d-42cc-b597-528a091feac9" volumeName="kubernetes.io/projected/788f5892-601d-42cc-b597-528a091feac9-kube-api-access-n9z28" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367881    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="907912b8-9313-43b3-b324-e8498ea6519a" volumeName="kubernetes.io/projected/907912b8-9313-43b3-b324-e8498ea6519a-kube-api-access-5mq42" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367892    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="907912b8-9313-43b3-b324-e8498ea6519a" volumeName="kubernetes.io/secret/907912b8-9313-43b3-b324-e8498ea6519a-serving-cert" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367903    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b555cf82-5423-4c47-a004-ae399654683b" volumeName="kubernetes.io/empty-dir/b555cf82-5423-4c47-a004-ae399654683b-tmp-dir" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367913    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf6ac89-eb35-4d19-a28c-6f879f02703a" volumeName="kubernetes.io/projected/cbf6ac89-eb35-4d19-a28c-6f879f02703a-kube-api-access-h4n7t" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367925    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d18cc860-7cd8-48d9-9a7a-40a40a53f379" volumeName="kubernetes.io/projected/d18cc860-7cd8-48d9-9a7a-40a40a53f379-kube-api-access-ppk4b" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367936    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67dad9d6-9671-49f3-84a2-024463451062" volumeName="kubernetes.io/projected/67dad9d6-9671-49f3-84a2-024463451062-kube-api-access-xq4h9" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367947    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7b0fed-186b-4d7b-affa-e91d5ef4b523" volumeName="kubernetes.io/secret/8e7b0fed-186b-4d7b-affa-e91d5ef4b523-agent-certs" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367962    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc9a426a-587d-4611-9298-90fc4b4f1dc3" volumeName="kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367973    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d25c4cc-371f-4a3d-8f68-b3d8923f0c67" volumeName="kubernetes.io/projected/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-kube-api-access-6mlbm" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367984    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d2109e3-d762-4bf9-bf3a-d405148ec32c" volumeName="kubernetes.io/empty-dir/0d2109e3-d762-4bf9-bf3a-d405148ec32c-data-volume" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.367995    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="343419d3-210d-4c12-86ff-c92f8f574b27" volumeName="kubernetes.io/empty-dir/343419d3-210d-4c12-86ff-c92f8f574b27-ca-trust-extracted" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368005    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395d5cbd-c077-4578-b377-3034e42a2b94" volumeName="kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-installation-pull-secrets" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368016    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c692538-d934-46cf-8c4c-d6a888dd00a1" volumeName="kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368027    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b" volumeName="kubernetes.io/projected/4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b-kube-api-access-28tfx" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368038    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e6da480-8072-42d2-98f3-91566744aa96" volumeName="kubernetes.io/projected/9e6da480-8072-42d2-98f3-91566744aa96-kube-api-access-5mr28" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368050    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6fec406-0515-44b5-ae95-c2fbfe84b452" volumeName="kubernetes.io/configmap/c6fec406-0515-44b5-ae95-c2fbfe84b452-cni-binary-copy" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368066    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d41c0686-14d0-43c4-928e-e38b57021fb7" volumeName="kubernetes.io/projected/d41c0686-14d0-43c4-928e-e38b57021fb7-kube-api-access-bp4k9" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368098    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="343419d3-210d-4c12-86ff-c92f8f574b27" volumeName="kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-registry-certificates" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368110    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395d5cbd-c077-4578-b377-3034e42a2b94" volumeName="kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-bound-sa-token" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368122    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e68d10c-d217-40d3-945d-c87ccbd7434b" volumeName="kubernetes.io/projected/5e68d10c-d217-40d3-945d-c87ccbd7434b-kube-api-access-9fnjr" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368133    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67dad9d6-9671-49f3-84a2-024463451062" volumeName="kubernetes.io/empty-dir/67dad9d6-9671-49f3-84a2-024463451062-archives-path" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368144    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6fec406-0515-44b5-ae95-c2fbfe84b452" volumeName="kubernetes.io/configmap/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-daemon-config" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368165    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d41c0686-14d0-43c4-928e-e38b57021fb7" volumeName="kubernetes.io/configmap/d41c0686-14d0-43c4-928e-e38b57021fb7-service-ca-bundle" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368177    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe5ff28-0896-4f54-b5cc-5326a0d60752" volumeName="kubernetes.io/configmap/1fe5ff28-0896-4f54-b5cc-5326a0d60752-config" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368192    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41551065-0510-4289-a742-83987e916774" volumeName="kubernetes.io/configmap/41551065-0510-4289-a742-83987e916774-iptables-alerter-script" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368203    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="623e4b93-06e4-44f9-bd85-d9b663df4834" volumeName="kubernetes.io/projected/623e4b93-06e4-44f9-bd85-d9b663df4834-kube-api-access-6t2hj" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368215    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="730fcc37-b18e-46e3-aea5-a7fba60b7a8c" volumeName="kubernetes.io/projected/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-kube-api-access-7q52f" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368226    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e6da480-8072-42d2-98f3-91566744aa96" volumeName="kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368237    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d41c0686-14d0-43c4-928e-e38b57021fb7" volumeName="kubernetes.io/empty-dir/d41c0686-14d0-43c4-928e-e38b57021fb7-tmp" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368248    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d2109e3-d762-4bf9-bf3a-d405148ec32c" volumeName="kubernetes.io/projected/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-api-access-s9hcr" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368261    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe5ff28-0896-4f54-b5cc-5326a0d60752" volumeName="kubernetes.io/projected/1fe5ff28-0896-4f54-b5cc-5326a0d60752-kube-api-access-twx6b" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368272    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="343419d3-210d-4c12-86ff-c92f8f574b27" volumeName="kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-installation-pull-secrets" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368283    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="730fcc37-b18e-46e3-aea5-a7fba60b7a8c" volumeName="kubernetes.io/configmap/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-config-volume" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368294    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89f2993a-ed5d-488e-832d-bcba8ddd9706" volumeName="kubernetes.io/secret/89f2993a-ed5d-488e-832d-bcba8ddd9706-cert" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368305    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99e3f70c-6dd8-467e-a444-7692e30a76a4" volumeName="kubernetes.io/empty-dir/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-tuned" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368319    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf6ac89-eb35-4d19-a28c-6f879f02703a" volumeName="kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-env-overrides" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368330    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe5ff28-0896-4f54-b5cc-5326a0d60752" volumeName="kubernetes.io/secret/1fe5ff28-0896-4f54-b5cc-5326a0d60752-serving-cert" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368342    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395d5cbd-c077-4578-b377-3034e42a2b94" volumeName="kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-trusted-ca" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368353    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395d5cbd-c077-4578-b377-3034e42a2b94" volumeName="kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-registry-certificates" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368371    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="503b82e1-e56c-4c80-9d63-fa8e647c0991" volumeName="kubernetes.io/configmap/503b82e1-e56c-4c80-9d63-fa8e647c0991-telemetry-config" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368384    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72c9fb60-7d7b-46b7-8d02-0b51dd1d0221" volumeName="kubernetes.io/secret/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-serving-cert" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368396    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b752c121-5e44-4df2-9f17-97f0c899516b" volumeName="kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-default-certificate" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368408    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41551065-0510-4289-a742-83987e916774" volumeName="kubernetes.io/projected/41551065-0510-4289-a742-83987e916774-kube-api-access-4ntgd" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368421    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="730fcc37-b18e-46e3-aea5-a7fba60b7a8c" volumeName="kubernetes.io/empty-dir/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-tmp-dir" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368432    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395d5cbd-c077-4578-b377-3034e42a2b94" volumeName="kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-kube-api-access-99bzl" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368442    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d25c4cc-371f-4a3d-8f68-b3d8923f0c67" volumeName="kubernetes.io/configmap/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-signing-cabundle" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368453    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89f2993a-ed5d-488e-832d-bcba8ddd9706" volumeName="kubernetes.io/projected/89f2993a-ed5d-488e-832d-bcba8ddd9706-kube-api-access-9b9tt" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368465    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b555cf82-5423-4c47-a004-ae399654683b" volumeName="kubernetes.io/projected/b555cf82-5423-4c47-a004-ae399654683b-kube-api-access-5nfhp" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368477    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395d5cbd-c077-4578-b377-3034e42a2b94" volumeName="kubernetes.io/empty-dir/395d5cbd-c077-4578-b377-3034e42a2b94-ca-trust-extracted" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368488    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="788f5892-601d-42cc-b597-528a091feac9" volumeName="kubernetes.io/configmap/788f5892-601d-42cc-b597-528a091feac9-samples-operator-config" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368499    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99e3f70c-6dd8-467e-a444-7692e30a76a4" volumeName="kubernetes.io/empty-dir/99e3f70c-6dd8-467e-a444-7692e30a76a4-tmp" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368511    5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b752c121-5e44-4df2-9f17-97f0c899516b" volumeName="kubernetes.io/projected/b752c121-5e44-4df2-9f17-97f0c899516b-kube-api-access-n278c" seLinuxMountContext=""
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368523    5072 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 03:57:00.368830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.368531    5072 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 03:57:00.370478 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.369059    5072 factory.go:153] Registering CRI-O factory
Oct 30 03:57:00.370478 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.369094    5072 factory.go:223] Registration of the crio container factory successfully
Oct 30 03:57:00.370478 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.369147    5072 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 03:57:00.370478 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.369164    5072 factory.go:103] Registering Raw factory
Oct 30 03:57:00.370478 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.369179    5072 manager.go:1199] Started watching for new ooms in manager
Oct 30 03:57:00.370478 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.369474    5072 manager.go:319] Starting recovery of all containers
Oct 30 03:57:00.382637 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.382618    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 03:57:00.382941 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.382758    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 03:57:00.398364 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.398324    5072 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 03:57:00.399759 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.399735    5072 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 03:57:00.399855 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.399787    5072 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 03:57:00.399855 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.399811    5072 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 03:57:00.399954 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:00.399850    5072 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 03:57:00.411434 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.411416    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 03:57:00.411523 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.411444    5072 generic.go:356] "Generic (PLEG): container finished" podID="907912b8-9313-43b3-b324-e8498ea6519a" containerID="9bb9bd5f4ffffda1ea02732ce99b0847014da4bd97bc3c7228c3d5abeb0a2e15" exitCode=255
Oct 30 03:57:00.416147 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.416126    5072 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 03:57:00.417254 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.417229    5072 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:57:00.417344 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.417298    5072 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:57:00.426503 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.426481    5072 generic.go:356] "Generic (PLEG): container finished" podID="be906c4a667f460e0e8153479b6ef733" containerID="9ed1e03d302f1c5c4c59904fa3a9c88a3e8e60963b7a6273b82ec5a69c0365b1" exitCode=0
Oct 30 03:57:00.430368 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.430351    5072 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:57:00.430723 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.430705    5072 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:57:00.454761 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.454706    5072 generic.go:356] "Generic (PLEG): container finished" podID="9e6da480-8072-42d2-98f3-91566744aa96" containerID="7c80aef88f8f0c52c71371234c7c7f5c7f40ce1ed547052d09aa3ae0a01ba640" exitCode=0
Oct 30 03:57:00.454862 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.454761    5072 generic.go:356] "Generic (PLEG): container finished" podID="9e6da480-8072-42d2-98f3-91566744aa96" containerID="f9a57bcc89b8f883b874d2c45e18f1a0cc3361e85f3e3794e155d0d82ab7b8d7" exitCode=0
Oct 30 03:57:00.454862 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.454775    5072 generic.go:356] "Generic (PLEG): container finished" podID="9e6da480-8072-42d2-98f3-91566744aa96" containerID="2b616797926ac77fd542be14e0379d36d053bfcb4d2c572d834faea1fd9e33ae" exitCode=0
Oct 30 03:57:00.454862 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.454787    5072 generic.go:356] "Generic (PLEG): container finished" podID="9e6da480-8072-42d2-98f3-91566744aa96" containerID="ed1b70edab7f9941b8e230d418a8100d3a89afc88de05dd9e3e8a845046c7a67" exitCode=0
Oct 30 03:57:00.454862 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.454796    5072 generic.go:356] "Generic (PLEG): container finished" podID="9e6da480-8072-42d2-98f3-91566744aa96" containerID="fadc70239664b552305d3c91e36bf1b9b181bf5be2e6048d7c23529e315eb36d" exitCode=0
Oct 30 03:57:00.454862 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.454806    5072 generic.go:356] "Generic (PLEG): container finished" podID="9e6da480-8072-42d2-98f3-91566744aa96" containerID="2d30ee5d6ee7dec553ea9de36e3e7b99f058393972d84c3b07d1da7981303f09" exitCode=0
Oct 30 03:57:00.468443 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.468428    5072 manager.go:324] Recovery completed
Oct 30 03:57:00.488579 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488556    5072 cpu_manager.go:226] "Starting" policy="none"
Oct 30 03:57:00.488579 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488571    5072 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 03:57:00.488682 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488587    5072 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 03:57:00.488724 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488702    5072 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 03:57:00.488753 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488713    5072 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 03:57:00.488753 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488729    5072 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 03:57:00.488753 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488735    5072 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 03:57:00.488753 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488741    5072 policy_none.go:50] "Start"
Oct 30 03:57:00.488753 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488751    5072 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 03:57:00.488879 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488763    5072 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 03:57:00.488909 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488888    5072 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 03:57:00.488909 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488895    5072 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 03:57:00.488909 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.488901    5072 policy_none.go:44] "Start"
Oct 30 03:57:00.493153 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.493128    5072 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 03:57:00.495697 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.495686    5072 manager.go:341] "Starting Device Plugin manager"
Oct 30 03:57:00.496168 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:00.495707    5072 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 03:57:00.496168 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.496163    5072 server.go:87] "Starting device plugin registration server"
Oct 30 03:57:00.496409 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.496399    5072 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 03:57:00.496447 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.496410    5072 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 03:57:00.496540 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.496517    5072 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 03:57:00.496640 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.496605    5072 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 03:57:00.496754 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.496643    5072 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 03:57:00.496754 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.496652    5072 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 03:57:00.496754 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.496681    5072 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T03:57:00.496632796Z","UUID":"f67bc712-8019-4c76-a6c2-0bb68efaa27d","Handler":null,"Name":"","Endpoint":""}
Oct 30 03:57:00.498000 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.497984    5072 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 03:57:00.498098 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.498008    5072 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 03:57:00.498181 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:00.498161    5072 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 03:57:00.499944 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.499909    5072 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"]
Oct 30 03:57:00.500470 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.500444    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal" event={"ID":"f95db1d51f36d0a32fb67f82ccd47c76","Type":"ContainerStarted","Data":"01f2ebbcd6a8f15c546c36f0f801ebc4b4a86b09460cc4a9d4a9c9e65d59f048"}
Oct 30 03:57:00.500543 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.500477    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal" event={"ID":"f95db1d51f36d0a32fb67f82ccd47c76","Type":"ContainerStarted","Data":"71b0828b77c7b2bc8ef6dae22ed644fe424ae384c7d887b6879527b1c4e121f5"}
Oct 30 03:57:00.500543 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.500519    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal" event={"ID":"be906c4a667f460e0e8153479b6ef733","Type":"ContainerStarted","Data":"225e3cd06f922c77363973c49c4d5e18ad69d3eb36c455f997b6bf307969812d"}
Oct 30 03:57:00.500543 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.500530    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal" event={"ID":"be906c4a667f460e0e8153479b6ef733","Type":"ContainerDied","Data":"9ed1e03d302f1c5c4c59904fa3a9c88a3e8e60963b7a6273b82ec5a69c0365b1"}
Oct 30 03:57:00.500543 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.500541    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal" event={"ID":"be906c4a667f460e0e8153479b6ef733","Type":"ContainerStarted","Data":"71809b6f9c61f215314c5d4a89b2bd5059bbaf724679fa389a0239400a4e0e41"}
Oct 30 03:57:00.500754 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.500743    5072 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.501068 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.501057    5072 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.527001 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.526981    5072 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:57:00.527100 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:00.527026    5072 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-99.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.527596 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.527586    5072 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:57:00.527643 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:00.527608    5072 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.596865 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.596811    5072 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:57:00.598146 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.598130    5072 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:57:00.598245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.598162    5072 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:57:00.598245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.598175    5072 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:57:00.598339 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.598248    5072 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.618275 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.618245    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f95db1d51f36d0a32fb67f82ccd47c76-config\") pod \"kube-apiserver-proxy-ip-10-0-139-99.ec2.internal\" (UID: \"f95db1d51f36d0a32fb67f82ccd47c76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.618275 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.618272    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be906c4a667f460e0e8153479b6ef733-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"be906c4a667f460e0e8153479b6ef733\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.618418 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.618290    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be906c4a667f460e0e8153479b6ef733-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"be906c4a667f460e0e8153479b6ef733\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.636071 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.636053    5072 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.636173 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.636119    5072 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.718601 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.718578    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be906c4a667f460e0e8153479b6ef733-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"be906c4a667f460e0e8153479b6ef733\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.718678 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.718605    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f95db1d51f36d0a32fb67f82ccd47c76-config\") pod \"kube-apiserver-proxy-ip-10-0-139-99.ec2.internal\" (UID: \"f95db1d51f36d0a32fb67f82ccd47c76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.718678 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.718620    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be906c4a667f460e0e8153479b6ef733-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"be906c4a667f460e0e8153479b6ef733\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.718678 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.718655    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be906c4a667f460e0e8153479b6ef733-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"be906c4a667f460e0e8153479b6ef733\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.718780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.718682    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be906c4a667f460e0e8153479b6ef733-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"be906c4a667f460e0e8153479b6ef733\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:00.718780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:00.718694    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f95db1d51f36d0a32fb67f82ccd47c76-config\") pod \"kube-apiserver-proxy-ip-10-0-139-99.ec2.internal\" (UID: \"f95db1d51f36d0a32fb67f82ccd47c76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Oct 30 03:57:01.348425 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.348390    5072 apiserver.go:52] "Watching apiserver"
Oct 30 03:57:01.416607 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.416576    5072 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 03:57:01.417408 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.417382    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bbcfd588b-rlkpf","openshift-image-registry/image-registry-759fbb47f9-s2s2f","openshift-multus/multus-ttzr9","openshift-network-console/networking-console-plugin-86c478bb88-htskn","openshift-ingress-canary/ingress-canary-bh464","openshift-network-diagnostics/network-check-target-k2wgk","openshift-service-ca/service-ca-5fffc97bdf-qc52w","openshift-console-operator/console-operator-7cd56b8dbc-664tl","openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7","openshift-multus/network-metrics-daemon-b67qk","openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn","openshift-monitoring/cluster-monitoring-operator-884576c57-sb794","kube-system/konnectivity-agent-d7nsr","openshift-dns/node-resolver-6njd4","openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d","openshift-insights/insights-operator-7c4986d449-trv7j","kube-system/global-pull-secret-syncer-d67gd","kube-system/kas-connection-checker-685895c7c9-7tc9g","kube-system/kas-connection-checker-685895c7c9-drvrv","openshift-multus/multus-additional-cni-plugins-2gkl8","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2","openshift-insights/periodic-gathering-qddvs-q47fm","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal","openshift-ingress/router-default-84bd8c5fd7-885hp","openshift-insights/insights-runtime-extractor-7x92g","kube-system/kas-connection-checker-685895c7c9-jmlk2","openshift-cluster-node-tuning-operator/tuned-qz4zz","openshift-dns/dns-default-xkq6m","openshift-image-registry/node-ca-mc77t","openshift-network-operator/iptables-alerter-6czxj","openshift-network-diagnostics/network-check-source-6f78d59f99-d29md","kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-c6hqf"]
Oct 30 03:57:01.418687 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.418646    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.419885 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.419866    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:57:01.421120 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.421101    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.421931 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.421914    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:57:01.422761 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.422743    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:57:01.429254 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.429235    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:57:01.433322 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.433303    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.434612 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.434570    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:01.435270 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.435251    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.436648 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.436624    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 03:57:01.436767 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.436751    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 03:57:01.437039 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.437023    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 03:57:01.437212 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.437197    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.437279 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.437265    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.438475 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.438459    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 03:57:01.440500 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.440484    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.440568 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.440538    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 03:57:01.440737 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.440721    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 03:57:01.440792 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.440750    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 03:57:01.440792 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.440754    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 03:57:01.440891 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.440877    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.440964 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.440888    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.440964 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.440898    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.441062 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.440995    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.441062 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.441034    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.449592 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.449576    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.451653 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.451637    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.451733 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.451669    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.451733 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.451693    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 03:57:01.451733 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.451715    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 03:57:01.451836 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.451804    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 03:57:01.451836 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.451804    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 03:57:01.453531 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.453511    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 03:57:01.453631 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.453560    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 03:57:01.453631 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.453567    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 03:57:01.453631 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.453560    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.453796 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.453632    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 03:57:01.453796 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.453642    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 03:57:01.453796 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.453680    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 03:57:01.453955 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.453938    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.454017 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.453992    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 03:57:01.454066 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454025    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.454150 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454113    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 03:57:01.454267 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454251    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.454327 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454286    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 03:57:01.454327 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454312    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.454409 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454339    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.454409 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454371    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 03:57:01.454595 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454575    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.454653 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454602    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.454713 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454689    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 03:57:01.454713 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454709    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 03:57:01.454805 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454751    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.454805 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454775    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.454805 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454782    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 03:57:01.454937 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454852    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 03:57:01.454937 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.454872    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 03:57:01.455041 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.455019    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 03:57:01.455116 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.455023    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.455939 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.455919    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.456270 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.456254    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vm8lh\""
Oct 30 03:57:01.456355 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.456309    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.456424 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.456374    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.456475 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.456430    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dnvhh\""
Oct 30 03:57:01.456679 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.456662    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kwf82\""
Oct 30 03:57:01.456963 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.456947    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-n6x98\""
Oct 30 03:57:01.457224 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.457206    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.457398 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.457379    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h7p9v\""
Oct 30 03:57:01.457620 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.457600    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.457793 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.457769    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.457868 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.457815    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.457922 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.457912    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 03:57:01.458301 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.458280    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 03:57:01.458414 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.458399    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.458545 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.458514    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.459022 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.459004    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 03:57:01.459155 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.459142    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.459228 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.459152    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 03:57:01.459228 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.459222    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.459330 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.459314    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.459385 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.459343    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.459451 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.459429    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 03:57:01.459507 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.459491    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 03:57:01.459727 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.459690    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w4blj\""
Oct 30 03:57:01.459843 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.459827    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-smvt7\""
Oct 30 03:57:01.459912 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.459897    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 03:57:01.459983 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.459968    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 03:57:01.460160 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.460142    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 03:57:01.460279 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.460263    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mwxht\""
Oct 30 03:57:01.460388 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.460267    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vnctk\""
Oct 30 03:57:01.460388 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.460353    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.460648 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.460618    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4xznk\""
Oct 30 03:57:01.460733 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.460649    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 03:57:01.460810 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.460786    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 03:57:01.460919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.460905    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.460999 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.460988    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8wd8q\""
Oct 30 03:57:01.461254 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.461238    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j9mnz\""
Oct 30 03:57:01.461513 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.461495    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rs54w\""
Oct 30 03:57:01.461905 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.461884    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b2qtm\""
Oct 30 03:57:01.461992 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.461940    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 03:57:01.461992 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.461952    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-846td\""
Oct 30 03:57:01.461992 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.461982    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hmk7j\""
Oct 30 03:57:01.462134 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.461955    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-skmdf\""
Oct 30 03:57:01.462134 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.462062    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 03:57:01.465511 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.465490    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 03:57:01.465511 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.465498    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 03:57:01.465636 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.465513    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tsnxp\""
Oct 30 03:57:01.465636 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.465544    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xpsj8\""
Oct 30 03:57:01.465636 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.465555    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 03:57:01.465636 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.465512    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 03:57:01.466327 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.466311    5072 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 03:57:01.466845 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.466828    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 03:57:01.466926 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.466835    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.466926 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.466895    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 03:57:01.466989 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.466955    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 03:57:01.522060 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522032    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jq7h\" (UniqueName: \"kubernetes.io/projected/5a290674-68db-4847-b2a1-ad968844f699-kube-api-access-6jq7h\") pod \"migrator-66df5d496b-2nlx7\" (UID: \"5a290674-68db-4847-b2a1-ad968844f699\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7"
Oct 30 03:57:01.522225 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522071    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppk4b\" (UniqueName: \"kubernetes.io/projected/d18cc860-7cd8-48d9-9a7a-40a40a53f379-kube-api-access-ppk4b\") pod \"kas-connection-checker-685895c7c9-7tc9g\" (UID: \"d18cc860-7cd8-48d9-9a7a-40a40a53f379\") " pod="kube-system/kas-connection-checker-685895c7c9-7tc9g"
Oct 30 03:57:01.522225 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522109    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41551065-0510-4289-a742-83987e916774-iptables-alerter-script\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:57:01.522225 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522129    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.522371 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522247    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-sys-fs\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.522371 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522264    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysctl-conf\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.522371 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522277    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-var-lib-kubelet\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.522371 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522324    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6fec406-0515-44b5-ae95-c2fbfe84b452-cni-binary-copy\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.522563 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522375    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-daemon-config\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.522563 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522472    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/395d5cbd-c077-4578-b377-3034e42a2b94-ca-trust-extracted\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.522563 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522529    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41551065-0510-4289-a742-83987e916774-host-slash\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:57:01.522711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522589    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-registration-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.522711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522597    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/395d5cbd-c077-4578-b377-3034e42a2b94-ca-trust-extracted\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.522711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522611    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41551065-0510-4289-a742-83987e916774-iptables-alerter-script\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:57:01.522711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522623    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b555cf82-5423-4c47-a004-ae399654683b-hosts-file\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:57:01.522711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522637    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6fec406-0515-44b5-ae95-c2fbfe84b452-cni-binary-copy\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.522711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522644    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-daemon-config\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.522711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522660    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t2hj\" (UniqueName: \"kubernetes.io/projected/623e4b93-06e4-44f9-bd85-d9b663df4834-kube-api-access-6t2hj\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:57:01.523015 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522724    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-hostroot\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.523015 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522746    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-signing-cabundle\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:57:01.523015 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522763    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-cni-bin\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.523015 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522790    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/343419d3-210d-4c12-86ff-c92f8f574b27-ca-trust-extracted\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.523015 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522866    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp4k9\" (UniqueName: \"kubernetes.io/projected/d41c0686-14d0-43c4-928e-e38b57021fb7-kube-api-access-bp4k9\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:57:01.523015 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522877    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/343419d3-210d-4c12-86ff-c92f8f574b27-ca-trust-extracted\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.523015 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522898    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89f2993a-ed5d-488e-832d-bcba8ddd9706-cert\") pod \"ingress-canary-bh464\" (UID: \"89f2993a-ed5d-488e-832d-bcba8ddd9706\") " pod="openshift-ingress-canary/ingress-canary-bh464"
Oct 30 03:57:01.523015 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522920    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-cnibin\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.523015 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522942    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2qhn\" (UniqueName: \"kubernetes.io/projected/491c21e4-c0e9-4d60-a00e-c896bcda29f7-kube-api-access-m2qhn\") pod \"migrator-66df5d496b-96bwk\" (UID: \"491c21e4-c0e9-4d60-a00e-c896bcda29f7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk"
Oct 30 03:57:01.523015 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522952    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-signing-cabundle\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:57:01.523015 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.522972    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d41c0686-14d0-43c4-928e-e38b57021fb7-service-ca-bundle\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:57:01.523015 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523013    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78q6r\" (UniqueName: \"kubernetes.io/projected/99e3f70c-6dd8-467e-a444-7692e30a76a4-kube-api-access-78q6r\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523108    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-installation-pull-secrets\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523132    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89f2993a-ed5d-488e-832d-bcba8ddd9706-cert\") pod \"ingress-canary-bh464\" (UID: \"89f2993a-ed5d-488e-832d-bcba8ddd9706\") " pod="openshift-ingress-canary/ingress-canary-bh464"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523135    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523198    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-config\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523330    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-config\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523328    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523322    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-trusted-ca\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523372    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-installation-pull-secrets\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523390    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-trusted-ca-bundle\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523417    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-log-socket\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523444    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523472    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523520    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-node-log\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.523570 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523563    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mr28\" (UniqueName: \"kubernetes.io/projected/9e6da480-8072-42d2-98f3-91566744aa96-kube-api-access-5mr28\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523701    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-netns\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523756    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-trusted-ca\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523784    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-registry-certificates\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523840    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523863    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523886    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktnq\" (UniqueName: \"kubernetes.io/projected/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-kube-api-access-cktnq\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523930    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.523959    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-tmp-dir\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524106    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-tmp-dir\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524120    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d41c0686-14d0-43c4-928e-e38b57021fb7-service-ca-bundle\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524126    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524169    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovnkube-script-lib\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524202    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99bzl\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-kube-api-access-99bzl\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.524253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524242    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-image-registry-private-configuration\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524284    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-trusted-ca-bundle\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524425    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovnkube-script-lib\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524449    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-image-registry-private-configuration\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524459    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq4h9\" (UniqueName: \"kubernetes.io/projected/67dad9d6-9671-49f3-84a2-024463451062-kube-api-access-xq4h9\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524484    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1fe5ff28-0896-4f54-b5cc-5326a0d60752-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524501    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twx6b\" (UniqueName: \"kubernetes.io/projected/1fe5ff28-0896-4f54-b5cc-5326a0d60752-kube-api-access-twx6b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524517    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-systemd\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524550    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovn-node-metrics-cert\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524570    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-socket-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524584    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-device-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524601    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpt9x\" (UniqueName: \"kubernetes.io/projected/0384e218-1937-4099-8566-67775cda0521-kube-api-access-lpt9x\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524616    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-kubernetes\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524636    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-system-cni-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524641    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-registry-certificates\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524655    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-cni-bin\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524670    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-cni-multus\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524681    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1fe5ff28-0896-4f54-b5cc-5326a0d60752-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524817    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28tfx\" (UniqueName: \"kubernetes.io/projected/4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b-kube-api-access-28tfx\") pod \"volume-data-source-validator-58c9f5f46d-cp4k2\" (UID: \"4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2"
Oct 30 03:57:01.524913 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524918    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-systemd\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524946    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovn-node-metrics-cert\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.524978    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525049    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tnhw\" (UniqueName: \"kubernetes.io/projected/1d40d8b3-4844-4a76-b395-b5b8a2a41438-kube-api-access-6tnhw\") pod \"network-check-source-6f78d59f99-d29md\" (UID: \"1d40d8b3-4844-4a76-b395-b5b8a2a41438\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-d29md"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525097    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-socket-dir-parent\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525155    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525209    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d2109e3-d762-4bf9-bf3a-d405148ec32c-data-volume\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525244    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-etc-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525298    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d2109e3-d762-4bf9-bf3a-d405148ec32c-data-volume\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525304    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovnkube-config\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525328    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-env-overrides\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525354    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525373    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27j9s\" (UniqueName: \"kubernetes.io/projected/503b82e1-e56c-4c80-9d63-fa8e647c0991-kube-api-access-27j9s\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525431    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-ovnkube-config\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525439    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-conf-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525472    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbf6ac89-eb35-4d19-a28c-6f879f02703a-env-overrides\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525476    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525503    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525523    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525546    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525571    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-modprobe-d\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525603    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99e3f70c-6dd8-467e-a444-7692e30a76a4-tmp\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.525626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525620    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/907912b8-9313-43b3-b324-e8498ea6519a-config\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525681    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99e3f70c-6dd8-467e-a444-7692e30a76a4-tmp\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525733    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mq42\" (UniqueName: \"kubernetes.io/projected/907912b8-9313-43b3-b324-e8498ea6519a-kube-api-access-5mq42\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525743    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/907912b8-9313-43b3-b324-e8498ea6519a-config\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525764    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-295lb\" (UniqueName: \"kubernetes.io/projected/478b6cf0-d0dd-4d65-aafb-ff211b60c188-kube-api-access-295lb\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525801    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525820    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e7b0fed-186b-4d7b-affa-e91d5ef4b523-agent-certs\") pod \"konnectivity-agent-d7nsr\" (UID: \"8e7b0fed-186b-4d7b-affa-e91d5ef4b523\") " pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525835    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-kubelet\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525851    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525870    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525899    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-multus-certs\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525918    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-signing-key\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.525964    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-kubelet\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526036    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e6da480-8072-42d2-98f3-91566744aa96-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526050    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-cni-netd\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526148    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-cnibin\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526155    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-signing-key\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526182    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e7b0fed-186b-4d7b-affa-e91d5ef4b523-konnectivity-ca\") pod \"konnectivity-agent-d7nsr\" (UID: \"8e7b0fed-186b-4d7b-affa-e91d5ef4b523\") " pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526212    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-serving-cert\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526235    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e7b0fed-186b-4d7b-affa-e91d5ef4b523-agent-certs\") pod \"konnectivity-agent-d7nsr\" (UID: \"8e7b0fed-186b-4d7b-affa-e91d5ef4b523\") " pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526277    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0384e218-1937-4099-8566-67775cda0521-serviceca\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526312    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526332    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n278c\" (UniqueName: \"kubernetes.io/projected/b752c121-5e44-4df2-9f17-97f0c899516b-kube-api-access-n278c\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526338    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-serving-cert\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526373    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e7b0fed-186b-4d7b-affa-e91d5ef4b523-konnectivity-ca\") pod \"konnectivity-agent-d7nsr\" (UID: \"8e7b0fed-186b-4d7b-affa-e91d5ef4b523\") " pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526444    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/907912b8-9313-43b3-b324-e8498ea6519a-serving-cert\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:57:01.526510 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526471    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-bound-sa-token\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526591    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9tt\" (UniqueName: \"kubernetes.io/projected/89f2993a-ed5d-488e-832d-bcba8ddd9706-kube-api-access-9b9tt\") pod \"ingress-canary-bh464\" (UID: \"89f2993a-ed5d-488e-832d-bcba8ddd9706\") " pod="openshift-ingress-canary/ingress-canary-bh464"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526629    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-var-lib-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526655    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-sys\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526777    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-cni-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526788    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0384e218-1937-4099-8566-67775cda0521-serviceca\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526815    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d2109e3-d762-4bf9-bf3a-d405148ec32c-crio-socket\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526846    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/907912b8-9313-43b3-b324-e8498ea6519a-serving-cert\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526841    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-systemd-units\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526878    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-system-cni-dir\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526918    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b555cf82-5423-4c47-a004-ae399654683b-tmp-dir\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526944    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1fe5ff28-0896-4f54-b5cc-5326a0d60752-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526970    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9hcr\" (UniqueName: \"kubernetes.io/projected/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-api-access-s9hcr\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.526998    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-default-certificate\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527001    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b555cf82-5423-4c47-a004-ae399654683b-tmp-dir\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527033    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-slash\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527059    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-run-ovn-kubernetes\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527105    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-installation-pull-secrets\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527153    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1fe5ff28-0896-4f54-b5cc-5326a0d60752-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527158    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d41c0686-14d0-43c4-928e-e38b57021fb7-tmp\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527191    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmnr\" (UniqueName: \"kubernetes.io/projected/4455e2df-ec71-4748-8920-36b2374af2a1-kube-api-access-gxmnr\") pod \"kas-connection-checker-685895c7c9-jmlk2\" (UID: \"4455e2df-ec71-4748-8920-36b2374af2a1\") " pod="kube-system/kas-connection-checker-685895c7c9-jmlk2"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527214    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-nginx-conf\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527228    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d41c0686-14d0-43c4-928e-e38b57021fb7-tmp\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527230    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-default-certificate\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.527245 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527231    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9z28\" (UniqueName: \"kubernetes.io/projected/788f5892-601d-42cc-b597-528a091feac9-kube-api-access-n9z28\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527263    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc9a426a-587d-4611-9298-90fc4b4f1dc3-kubelet-config\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527286    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-host\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527311    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-installation-pull-secrets\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527313    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-image-registry-private-configuration\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527378    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntgd\" (UniqueName: \"kubernetes.io/projected/41551065-0510-4289-a742-83987e916774-kube-api-access-4ntgd\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527409    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nfhp\" (UniqueName: \"kubernetes.io/projected/b555cf82-5423-4c47-a004-ae399654683b-kube-api-access-5nfhp\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527417    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-nginx-conf\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527496    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-image-registry-private-configuration\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527525    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/67dad9d6-9671-49f3-84a2-024463451062-archives-path\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527549    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4n7t\" (UniqueName: \"kubernetes.io/projected/cbf6ac89-eb35-4d19-a28c-6f879f02703a-kube-api-access-h4n7t\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527579    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/67dad9d6-9671-49f3-84a2-024463451062-archives-path\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527608    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/788f5892-601d-42cc-b597-528a091feac9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527626    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-run\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527644    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysconfig\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527684    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q52f\" (UniqueName: \"kubernetes.io/projected/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-kube-api-access-7q52f\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527727    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-k8s-cni-cncf-io\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527823    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/788f5892-601d-42cc-b597-528a091feac9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527843    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527874    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-ovn\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527897    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d41c0686-14d0-43c4-928e-e38b57021fb7-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:57:01.528046 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.527942    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d41c0686-14d0-43c4-928e-e38b57021fb7-serving-cert\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528145    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0384e218-1937-4099-8566-67775cda0521-host\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528216    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysctl-d\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528249    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-trusted-ca\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528282    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-bound-sa-token\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528342    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-lib-modules\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528379    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-tuned\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528414    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlbm\" (UniqueName: \"kubernetes.io/projected/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-kube-api-access-6mlbm\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528416    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d41c0686-14d0-43c4-928e-e38b57021fb7-serving-cert\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528483    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fnjr\" (UniqueName: \"kubernetes.io/projected/5e68d10c-d217-40d3-945d-c87ccbd7434b-kube-api-access-9fnjr\") pod \"kas-connection-checker-685895c7c9-drvrv\" (UID: \"5e68d10c-d217-40d3-945d-c87ccbd7434b\") " pod="kube-system/kas-connection-checker-685895c7c9-drvrv"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528513    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-tuned\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528520    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-stats-auth\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528553    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-run-netns\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528584    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8xt\" (UniqueName: \"kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt\") pod \"network-check-target-k2wgk\" (UID: \"4c692538-d934-46cf-8c4c-d6a888dd00a1\") " pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528614    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc9a426a-587d-4611-9298-90fc4b4f1dc3-dbus\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528640    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528676    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d41c0686-14d0-43c4-928e-e38b57021fb7-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528721    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-etc-kubernetes\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.528780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528744    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-stats-auth\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528803    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcl9n\" (UniqueName: \"kubernetes.io/projected/c6fec406-0515-44b5-ae95-c2fbfe84b452-kube-api-access-qcl9n\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528844    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/907912b8-9313-43b3-b324-e8498ea6519a-trusted-ca\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528878    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-registry-certificates\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528908    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-service-ca-bundle\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528922    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc9a426a-587d-4611-9298-90fc4b4f1dc3-original-pull-secret\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528938    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdlq\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-kube-api-access-vzdlq\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528965    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d41c0686-14d0-43c4-928e-e38b57021fb7-snapshots\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.528992    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/503b82e1-e56c-4c80-9d63-fa8e647c0991-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.529019    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-config-volume\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.529047    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-trusted-ca\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.529133    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/907912b8-9313-43b3-b324-e8498ea6519a-trusted-ca\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.529217    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d41c0686-14d0-43c4-928e-e38b57021fb7-snapshots\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.529302    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-config-volume\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.529365    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/503b82e1-e56c-4c80-9d63-fa8e647c0991-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:57:01.529547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.529413    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-registry-certificates\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.529930 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.529619    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-service-ca-bundle\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:57:01.547026 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.546999    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t2hj\" (UniqueName: \"kubernetes.io/projected/623e4b93-06e4-44f9-bd85-d9b663df4834-kube-api-access-6t2hj\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:57:01.547026 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.547018    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mr28\" (UniqueName: \"kubernetes.io/projected/9e6da480-8072-42d2-98f3-91566744aa96-kube-api-access-5mr28\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.547225 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.547060    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jq7h\" (UniqueName: \"kubernetes.io/projected/5a290674-68db-4847-b2a1-ad968844f699-kube-api-access-6jq7h\") pod \"migrator-66df5d496b-2nlx7\" (UID: \"5a290674-68db-4847-b2a1-ad968844f699\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2nlx7"
Oct 30 03:57:01.547225 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.547060    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp4k9\" (UniqueName: \"kubernetes.io/projected/d41c0686-14d0-43c4-928e-e38b57021fb7-kube-api-access-bp4k9\") pod \"insights-operator-7c4986d449-trv7j\" (UID: \"d41c0686-14d0-43c4-928e-e38b57021fb7\") " pod="openshift-insights/insights-operator-7c4986d449-trv7j"
Oct 30 03:57:01.547225 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.547131    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppk4b\" (UniqueName: \"kubernetes.io/projected/d18cc860-7cd8-48d9-9a7a-40a40a53f379-kube-api-access-ppk4b\") pod \"kas-connection-checker-685895c7c9-7tc9g\" (UID: \"d18cc860-7cd8-48d9-9a7a-40a40a53f379\") " pod="kube-system/kas-connection-checker-685895c7c9-7tc9g"
Oct 30 03:57:01.547392 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.547374    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99bzl\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-kube-api-access-99bzl\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.547473 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.547457    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2qhn\" (UniqueName: \"kubernetes.io/projected/491c21e4-c0e9-4d60-a00e-c896bcda29f7-kube-api-access-m2qhn\") pod \"migrator-66df5d496b-96bwk\" (UID: \"491c21e4-c0e9-4d60-a00e-c896bcda29f7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-96bwk"
Oct 30 03:57:01.547704 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.547688    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-295lb\" (UniqueName: \"kubernetes.io/projected/478b6cf0-d0dd-4d65-aafb-ff211b60c188-kube-api-access-295lb\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.547734 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.547700    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tnhw\" (UniqueName: \"kubernetes.io/projected/1d40d8b3-4844-4a76-b395-b5b8a2a41438-kube-api-access-6tnhw\") pod \"network-check-source-6f78d59f99-d29md\" (UID: \"1d40d8b3-4844-4a76-b395-b5b8a2a41438\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-d29md"
Oct 30 03:57:01.549695 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.549679    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cktnq\" (UniqueName: \"kubernetes.io/projected/72c9fb60-7d7b-46b7-8d02-0b51dd1d0221-kube-api-access-cktnq\") pod \"service-ca-operator-544d749c8c-b4n4d\" (UID: \"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d"
Oct 30 03:57:01.549813 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.549797    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27j9s\" (UniqueName: \"kubernetes.io/projected/503b82e1-e56c-4c80-9d63-fa8e647c0991-kube-api-access-27j9s\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:57:01.549958 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.549941    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twx6b\" (UniqueName: \"kubernetes.io/projected/1fe5ff28-0896-4f54-b5cc-5326a0d60752-kube-api-access-twx6b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx\" (UID: \"1fe5ff28-0896-4f54-b5cc-5326a0d60752\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx"
Oct 30 03:57:01.554928 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.554906    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:01.558396 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.557993    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpt9x\" (UniqueName: \"kubernetes.io/projected/0384e218-1937-4099-8566-67775cda0521-kube-api-access-lpt9x\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:57:01.558396 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.558057    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78q6r\" (UniqueName: \"kubernetes.io/projected/99e3f70c-6dd8-467e-a444-7692e30a76a4-kube-api-access-78q6r\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.558396 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.558065    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-bound-sa-token\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.558396 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.558139    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tfx\" (UniqueName: \"kubernetes.io/projected/4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b-kube-api-access-28tfx\") pod \"volume-data-source-validator-58c9f5f46d-cp4k2\" (UID: \"4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cp4k2"
Oct 30 03:57:01.558396 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.558323    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq4h9\" (UniqueName: \"kubernetes.io/projected/67dad9d6-9671-49f3-84a2-024463451062-kube-api-access-xq4h9\") pod \"periodic-gathering-qddvs-q47fm\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") " pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:57:01.558396 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.558362    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n278c\" (UniqueName: \"kubernetes.io/projected/b752c121-5e44-4df2-9f17-97f0c899516b-kube-api-access-n278c\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.558638 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.558552    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlbm\" (UniqueName: \"kubernetes.io/projected/7d25c4cc-371f-4a3d-8f68-b3d8923f0c67-kube-api-access-6mlbm\") pod \"service-ca-5fffc97bdf-qc52w\" (UID: \"7d25c4cc-371f-4a3d-8f68-b3d8923f0c67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qc52w"
Oct 30 03:57:01.558638 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.558573    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fnjr\" (UniqueName: \"kubernetes.io/projected/5e68d10c-d217-40d3-945d-c87ccbd7434b-kube-api-access-9fnjr\") pod \"kas-connection-checker-685895c7c9-drvrv\" (UID: \"5e68d10c-d217-40d3-945d-c87ccbd7434b\") " pod="kube-system/kas-connection-checker-685895c7c9-drvrv"
Oct 30 03:57:01.558638 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.558582    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmnr\" (UniqueName: \"kubernetes.io/projected/4455e2df-ec71-4748-8920-36b2374af2a1-kube-api-access-gxmnr\") pod \"kas-connection-checker-685895c7c9-jmlk2\" (UID: \"4455e2df-ec71-4748-8920-36b2374af2a1\") " pod="kube-system/kas-connection-checker-685895c7c9-jmlk2"
Oct 30 03:57:01.558733 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.558664    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nfhp\" (UniqueName: \"kubernetes.io/projected/b555cf82-5423-4c47-a004-ae399654683b-kube-api-access-5nfhp\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:57:01.558844 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.558822    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9tt\" (UniqueName: \"kubernetes.io/projected/89f2993a-ed5d-488e-832d-bcba8ddd9706-kube-api-access-9b9tt\") pod \"ingress-canary-bh464\" (UID: \"89f2993a-ed5d-488e-832d-bcba8ddd9706\") " pod="openshift-ingress-canary/ingress-canary-bh464"
Oct 30 03:57:01.558911 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.558845    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9hcr\" (UniqueName: \"kubernetes.io/projected/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-api-access-s9hcr\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.559014 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.558995    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntgd\" (UniqueName: \"kubernetes.io/projected/41551065-0510-4289-a742-83987e916774-kube-api-access-4ntgd\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:57:01.559236 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.559219    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mq42\" (UniqueName: \"kubernetes.io/projected/907912b8-9313-43b3-b324-e8498ea6519a-kube-api-access-5mq42\") pod \"console-operator-7cd56b8dbc-664tl\" (UID: \"907912b8-9313-43b3-b324-e8498ea6519a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:57:01.560758 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.560738    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr8xt\" (UniqueName: \"kubernetes.io/projected/4c692538-d934-46cf-8c4c-d6a888dd00a1-kube-api-access-cr8xt\") pod \"network-check-target-k2wgk\" (UID: \"4c692538-d934-46cf-8c4c-d6a888dd00a1\") " pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:57:01.561111 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.561061    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q52f\" (UniqueName: \"kubernetes.io/projected/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-kube-api-access-7q52f\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:01.561295 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.561278    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcl9n\" (UniqueName: \"kubernetes.io/projected/c6fec406-0515-44b5-ae95-c2fbfe84b452-kube-api-access-qcl9n\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.561408 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.561388    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9z28\" (UniqueName: \"kubernetes.io/projected/788f5892-601d-42cc-b597-528a091feac9-kube-api-access-n9z28\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:57:01.561489 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.561476    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdlq\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-kube-api-access-vzdlq\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.561540 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.561525    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4n7t\" (UniqueName: \"kubernetes.io/projected/cbf6ac89-eb35-4d19-a28c-6f879f02703a-kube-api-access-h4n7t\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.564198 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.564182    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-bound-sa-token\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.629736 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629654    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-systemd\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.629736 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629686    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-socket-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.629736 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629701    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-device-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.629736 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629715    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-kubernetes\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.629736 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629731    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-system-cni-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.630106 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629753    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-cni-bin\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.630106 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629775    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-cni-multus\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.630106 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629789    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-systemd\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.630106 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629792    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-device-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.630106 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629830    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-cni-multus\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.630106 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629831    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-cni-bin\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.630106 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629844    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-socket-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.630106 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629800    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-kubernetes\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.630468 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629801    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-systemd\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.630468 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.629808    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-system-cni-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.630468 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630259    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-systemd\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.630468 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630315    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-socket-dir-parent\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.630468 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630361    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-etc-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.630468 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630425    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-socket-dir-parent\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.630468 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630442    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:57:01.630776 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630474    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-conf-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.630776 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630497    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.630776 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630445    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-etc-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.630776 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630560    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-conf-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.630776 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.630580    5072 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 03:57:01.630776 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.630616    5072 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:57:01.630776 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630646    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.630776 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.630684    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs podName:623e4b93-06e4-44f9-bd85-d9b663df4834 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:02.130663091 +0000 UTC m=+1.872086185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs") pod "network-metrics-daemon-b67qk" (UID: "623e4b93-06e4-44f9-bd85-d9b663df4834") : secret "metrics-daemon-secret" not found
Oct 30 03:57:01.630776 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.630703    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert podName:01bc8bd9-f593-4acf-97c0-fc6c4beb4867 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:02.13069335 +0000 UTC m=+1.872116451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htskn" (UID: "01bc8bd9-f593-4acf-97c0-fc6c4beb4867") : secret "networking-console-plugin-cert" not found
Oct 30 03:57:01.630776 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630747    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:57:01.631253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630786    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:57:01.631253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630814    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:57:01.631253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630844    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-modprobe-d\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.631253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630878    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:01.631253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630910    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-kubelet\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.631253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630946    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:01.631253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.630970    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-multus-certs\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.631253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631101    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-kubelet\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.631253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631134    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-cni-netd\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.631253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631199    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-cnibin\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.631253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631238    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631268    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-var-lib-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631298    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-sys\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631326    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-cni-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631356    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d2109e3-d762-4bf9-bf3a-d405148ec32c-crio-socket\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631334    5072 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631385    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-systemd-units\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631419    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-system-cni-dir\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631454    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-slash\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631487    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-run-ovn-kubernetes\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631527    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc9a426a-587d-4611-9298-90fc4b4f1dc3-kubelet-config\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631598    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc9a426a-587d-4611-9298-90fc4b4f1dc3-kubelet-config\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631654    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-var-lib-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.631728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631713    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-sys\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.632297 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631774    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-multus-cni-dir\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.632297 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631830    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d2109e3-d762-4bf9-bf3a-d405148ec32c-crio-socket\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.632297 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631909    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-systemd-units\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.632297 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.631959    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-system-cni-dir\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.632297 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632001    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-slash\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.632297 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.632097    5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:57:01.632297 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632089    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-run-ovn-kubernetes\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.632297 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.632114    5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-759fbb47f9-s2s2f: secret "image-registry-tls" not found
Oct 30 03:57:01.632297 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.632184    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls podName:395d5cbd-c077-4578-b377-3034e42a2b94 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:02.132160805 +0000 UTC m=+1.873583902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls") pod "image-registry-759fbb47f9-s2s2f" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94") : secret "image-registry-tls" not found
Oct 30 03:57:01.632297 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632217    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-kubelet\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.632297 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632265    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-multus-certs\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632314    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-modprobe-d\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632327    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-var-lib-kubelet\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632394    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-cni-netd\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.632433    5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.632445    5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbcfd588b-rlkpf: secret "image-registry-tls" not found
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632444    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-cnibin\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.632522    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls podName:343419d3-210d-4c12-86ff-c92f8f574b27 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:02.132500696 +0000 UTC m=+1.873923778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls") pod "image-registry-6bbcfd588b-rlkpf" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27") : secret "image-registry-tls" not found
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632540    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-host\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632576    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-run\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632605    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysconfig\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632633    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-k8s-cni-cncf-io\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632657    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632687    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-ovn\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632716    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0384e218-1937-4099-8566-67775cda0521-host\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632744    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysctl-d\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.632784 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632771    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-lib-modules\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632807    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-run-netns\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632838    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc9a426a-587d-4611-9298-90fc4b4f1dc3-dbus\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632867    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-etc-kubernetes\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632905    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632927    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-sys-fs\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632955    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysctl-conf\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.632985    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-var-lib-kubelet\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633017    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41551065-0510-4289-a742-83987e916774-host-slash\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633044    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-registration-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633068    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b555cf82-5423-4c47-a004-ae399654683b-hosts-file\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633114    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-hostroot\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633144    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-cni-bin\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633176    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-cnibin\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633210    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-log-socket\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633238    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633269    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633299    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-node-log\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633332    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-netns\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633359    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633390    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633423    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.633505 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.633453    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.634475 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634058    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d2109e3-d762-4bf9-bf3a-d405148ec32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:01.634475 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634151    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-host\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.634475 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634202    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-run\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.634475 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634248    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysconfig\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.634475 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634294    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-k8s-cni-cncf-io\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.634475 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634340    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41551065-0510-4289-a742-83987e916774-host-slash\") pod \"iptables-alerter-6czxj\" (UID: \"41551065-0510-4289-a742-83987e916774\") " pod="openshift-network-operator/iptables-alerter-6czxj"
Oct 30 03:57:01.634475 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634348    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b555cf82-5423-4c47-a004-ae399654683b-hosts-file\") pod \"node-resolver-6njd4\" (UID: \"b555cf82-5423-4c47-a004-ae399654683b\") " pod="openshift-dns/node-resolver-6njd4"
Oct 30 03:57:01.634475 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634392    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-hostroot\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.634475 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634439    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-cni-bin\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.634863 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634493    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-cnibin\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.634863 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634538    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-log-socket\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.634863 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.634637    5072 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:57:01.634863 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.634683    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls podName:730fcc37-b18e-46e3-aea5-a7fba60b7a8c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:02.134669195 +0000 UTC m=+1.876092275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls") pod "dns-default-xkq6m" (UID: "730fcc37-b18e-46e3-aea5-a7fba60b7a8c") : secret "dns-default-metrics-tls" not found
Oct 30 03:57:01.634863 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.634757    5072 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:57:01.634863 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:01.634795    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls podName:0d2109e3-d762-4bf9-bf3a-d405148ec32c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:02.134784483 +0000 UTC m=+1.876207558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7x92g" (UID: "0d2109e3-d762-4bf9-bf3a-d405148ec32c") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:57:01.634863 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634838    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-node-log\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.635241 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634889    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-host-run-netns\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.635241 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634933    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.635241 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.634989    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e6da480-8072-42d2-98f3-91566744aa96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gkl8\" (UID: \"9e6da480-8072-42d2-98f3-91566744aa96\") " pod="openshift-multus/multus-additional-cni-plugins-2gkl8"
Oct 30 03:57:01.635241 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635043    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.635241 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635104    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6fec406-0515-44b5-ae95-c2fbfe84b452-etc-kubernetes\") pod \"multus-ttzr9\" (UID: \"c6fec406-0515-44b5-ae95-c2fbfe84b452\") " pod="openshift-multus/multus-ttzr9"
Oct 30 03:57:01.635241 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635151    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-ovn\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.635241 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635191    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0384e218-1937-4099-8566-67775cda0521-host\") pod \"node-ca-mc77t\" (UID: \"0384e218-1937-4099-8566-67775cda0521\") " pod="openshift-image-registry/node-ca-mc77t"
Oct 30 03:57:01.635542 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635244    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b752c121-5e44-4df2-9f17-97f0c899516b-service-ca-bundle\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.635542 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635299    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysctl-d\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.635542 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635321    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-run-openvswitch\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.635542 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635376    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-lib-modules\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.635542 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635388    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-sys-fs\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.635542 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635421    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbf6ac89-eb35-4d19-a28c-6f879f02703a-host-run-netns\") pod \"ovnkube-node-c6hqf\" (UID: \"cbf6ac89-eb35-4d19-a28c-6f879f02703a\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:01.635542 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635485    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc9a426a-587d-4611-9298-90fc4b4f1dc3-dbus\") pod \"global-pull-secret-syncer-d67gd\" (UID: \"bc9a426a-587d-4611-9298-90fc4b4f1dc3\") " pod="kube-system/global-pull-secret-syncer-d67gd"
Oct 30 03:57:01.635542 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635532    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-etc-sysctl-conf\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.635888 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635548    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99e3f70c-6dd8-467e-a444-7692e30a76a4-var-lib-kubelet\") pod \"tuned-qz4zz\" (UID: \"99e3f70c-6dd8-467e-a444-7692e30a76a4\") " pod="openshift-cluster-node-tuning-operator/tuned-qz4zz"
Oct 30 03:57:01.635888 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.635593    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/478b6cf0-d0dd-4d65-aafb-ff211b60c188-registration-dir\") pod \"aws-ebs-csi-driver-node-m8cpn\" (UID: \"478b6cf0-d0dd-4d65-aafb-ff211b60c188\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8cpn"
Oct 30 03:57:01.638107 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.638056    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/788f5892-601d-42cc-b597-528a091feac9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xccsz\" (UID: \"788f5892-601d-42cc-b597-528a091feac9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:57:01.638549 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.638523    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/503b82e1-e56c-4c80-9d63-fa8e647c0991-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sb794\" (UID: \"503b82e1-e56c-4c80-9d63-fa8e647c0991\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:57:01.638911 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.638895    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752c121-5e44-4df2-9f17-97f0c899516b-metrics-certs\") pod \"router-default-84bd8c5fd7-885hp\" (UID: \"b752c121-5e44-4df2-9f17-97f0c899516b\") " pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.720201 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.720173    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"
Oct 30 03:57:01.721475 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.721452    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:01.722356 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.722335    5072 scope.go:122] "RemoveContainer" containerID="9bb9bd5f4ffffda1ea02732ce99b0847014da4bd97bc3c7228c3d5abeb0a2e15"
Oct 30 03:57:01.722405 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.722393    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"
Oct 30 03:57:01.806909 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:01.806839    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:02.035759 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.035735    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz"]
Oct 30 03:57:02.041792 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.041766    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sb794"]
Oct 30 03:57:02.044699 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.044642    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:02.045806 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.045789    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84bd8c5fd7-885hp"]
Oct 30 03:57:02.049270 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:02.049238    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503b82e1_e56c_4c80_9d63_fa8e647c0991.slice/crio-55d0ef8b96b54d0c96105067844bc0e823190662028ceb37b8235530250787a1 WatchSource:0}: Error finding container 55d0ef8b96b54d0c96105067844bc0e823190662028ceb37b8235530250787a1: Status 404 returned error can't find the container with id 55d0ef8b96b54d0c96105067844bc0e823190662028ceb37b8235530250787a1
Oct 30 03:57:02.049542 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:02.049523    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb752c121_5e44_4df2_9f17_97f0c899516b.slice/crio-b5c494cb2c8d831d1d78a684c6518179a9c04f978d29b4038cf77315752e2b99 WatchSource:0}: Error finding container b5c494cb2c8d831d1d78a684c6518179a9c04f978d29b4038cf77315752e2b99: Status 404 returned error can't find the container with id b5c494cb2c8d831d1d78a684c6518179a9c04f978d29b4038cf77315752e2b99
Oct 30 03:57:02.051359 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.051346    5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:57:02.138144 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.138117    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:02.138262 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.138156    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:02.138314 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138275    5072 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:57:02.138314 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.138288    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:57:02.138371 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138321    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls podName:0d2109e3-d762-4bf9-bf3a-d405148ec32c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:03.138307068 +0000 UTC m=+2.879730143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7x92g" (UID: "0d2109e3-d762-4bf9-bf3a-d405148ec32c") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:57:02.138371 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.138339    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:57:02.138371 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138365    5072 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 03:57:02.138491 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.138372    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:02.138491 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.138391    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:02.138491 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138403    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs podName:623e4b93-06e4-44f9-bd85-d9b663df4834 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:03.138393084 +0000 UTC m=+2.879816159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs") pod "network-metrics-daemon-b67qk" (UID: "623e4b93-06e4-44f9-bd85-d9b663df4834") : secret "metrics-daemon-secret" not found
Oct 30 03:57:02.138491 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138444    5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:57:02.138491 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138456    5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-759fbb47f9-s2s2f: secret "image-registry-tls" not found
Oct 30 03:57:02.138743 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138514    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls podName:395d5cbd-c077-4578-b377-3034e42a2b94 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:03.138504202 +0000 UTC m=+2.879927277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls") pod "image-registry-759fbb47f9-s2s2f" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94") : secret "image-registry-tls" not found
Oct 30 03:57:02.138743 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138658    5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:57:02.138743 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138664    5072 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:57:02.138743 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138670    5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbcfd588b-rlkpf: secret "image-registry-tls" not found
Oct 30 03:57:02.138743 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138660    5072 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:57:02.138743 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138696    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert podName:01bc8bd9-f593-4acf-97c0-fc6c4beb4867 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:03.138688661 +0000 UTC m=+2.880111768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htskn" (UID: "01bc8bd9-f593-4acf-97c0-fc6c4beb4867") : secret "networking-console-plugin-cert" not found
Oct 30 03:57:02.138743 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138707    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls podName:343419d3-210d-4c12-86ff-c92f8f574b27 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:03.138702134 +0000 UTC m=+2.880125209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls") pod "image-registry-6bbcfd588b-rlkpf" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27") : secret "image-registry-tls" not found
Oct 30 03:57:02.138743 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:02.138718    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls podName:730fcc37-b18e-46e3-aea5-a7fba60b7a8c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:03.138711886 +0000 UTC m=+2.880134960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls") pod "dns-default-xkq6m" (UID: "730fcc37-b18e-46e3-aea5-a7fba60b7a8c") : secret "dns-default-metrics-tls" not found
Oct 30 03:57:02.347756 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.347681    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:02.409139 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.407826    5072 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:57:02.467484 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.467448    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz" event={"ID":"788f5892-601d-42cc-b597-528a091feac9","Type":"ContainerStarted","Data":"8ed1a574a2ae8c7518fb26e96ae9e188df6eddbb8362fcf16592cfbb0828570c"}
Oct 30 03:57:02.472183 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.472159    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 03:57:02.472351 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.472260    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl" event={"ID":"907912b8-9313-43b3-b324-e8498ea6519a","Type":"ContainerStarted","Data":"61781c9ff9450a59a3911332470d5702098becd7036d752ac4646f4040dabd52"}
Oct 30 03:57:02.474004 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.473978    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84bd8c5fd7-885hp" event={"ID":"b752c121-5e44-4df2-9f17-97f0c899516b","Type":"ContainerStarted","Data":"94f408def86aef7458bddd5936fd9b107bfde8b7549d9421457d2d7b87be2211"}
Oct 30 03:57:02.474129 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.474023    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84bd8c5fd7-885hp" event={"ID":"b752c121-5e44-4df2-9f17-97f0c899516b","Type":"ContainerStarted","Data":"b5c494cb2c8d831d1d78a684c6518179a9c04f978d29b4038cf77315752e2b99"}
Oct 30 03:57:02.475111 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.475067    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794" event={"ID":"503b82e1-e56c-4c80-9d63-fa8e647c0991","Type":"ContainerStarted","Data":"55d0ef8b96b54d0c96105067844bc0e823190662028ceb37b8235530250787a1"}
Oct 30 03:57:02.637506 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.637427    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:02.722580 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.722545    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:02.725477 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.725453    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:02.883389 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:02.883354    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:03.148893 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.148831    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:03.148893 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.148894    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:03.149140 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.148963    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:57:03.149140 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.148994    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:57:03.149140 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:03.149009    5072 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:57:03.149140 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:03.149016    5072 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:57:03.149140 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.149025    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:03.149140 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.149054    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:03.149140 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:03.149094    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls podName:0d2109e3-d762-4bf9-bf3a-d405148ec32c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:05.149058775 +0000 UTC m=+4.890481870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7x92g" (UID: "0d2109e3-d762-4bf9-bf3a-d405148ec32c") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:57:03.149140 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:03.149144    5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:57:03.149476 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:03.149155    5072 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:57:03.149476 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:03.149157    5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbcfd588b-rlkpf: secret "image-registry-tls" not found
Oct 30 03:57:03.149476 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:03.149203    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls podName:730fcc37-b18e-46e3-aea5-a7fba60b7a8c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:05.149189739 +0000 UTC m=+4.890612818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls") pod "dns-default-xkq6m" (UID: "730fcc37-b18e-46e3-aea5-a7fba60b7a8c") : secret "dns-default-metrics-tls" not found
Oct 30 03:57:03.149476 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:03.149220    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls podName:343419d3-210d-4c12-86ff-c92f8f574b27 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:05.149210847 +0000 UTC m=+4.890633925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls") pod "image-registry-6bbcfd588b-rlkpf" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27") : secret "image-registry-tls" not found
Oct 30 03:57:03.149476 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:03.149253    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert podName:01bc8bd9-f593-4acf-97c0-fc6c4beb4867 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:05.149232848 +0000 UTC m=+4.890655935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htskn" (UID: "01bc8bd9-f593-4acf-97c0-fc6c4beb4867") : secret "networking-console-plugin-cert" not found
Oct 30 03:57:03.149476 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:03.149278    5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:57:03.149476 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:03.149288    5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-759fbb47f9-s2s2f: secret "image-registry-tls" not found
Oct 30 03:57:03.149476 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:03.149316    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls podName:395d5cbd-c077-4578-b377-3034e42a2b94 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:05.14930734 +0000 UTC m=+4.890730443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls") pod "image-registry-759fbb47f9-s2s2f" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94") : secret "image-registry-tls" not found
Oct 30 03:57:03.151828 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.151798    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623e4b93-06e4-44f9-bd85-d9b663df4834-metrics-certs\") pod \"network-metrics-daemon-b67qk\" (UID: \"623e4b93-06e4-44f9-bd85-d9b663df4834\") " pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:57:03.157583 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.157533    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:03.230493 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.230467    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b67qk"
Oct 30 03:57:03.368595 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.368552    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:03.479407 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.479375    5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:57:03.532247 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.532214    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b67qk"]
Oct 30 03:57:03.539111 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:03.534327    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623e4b93_06e4_44f9_bd85_d9b663df4834.slice/crio-5bb9dedc69af52fd1e995ef87eb734be7b73e7f7bb8915c3f511589f0286525e WatchSource:0}: Error finding container 5bb9dedc69af52fd1e995ef87eb734be7b73e7f7bb8915c3f511589f0286525e: Status 404 returned error can't find the container with id 5bb9dedc69af52fd1e995ef87eb734be7b73e7f7bb8915c3f511589f0286525e
Oct 30 03:57:03.588494 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.588457    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:03.803051 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:03.803024    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:04.017905 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.017325    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:04.302112 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.302061    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:57:04.303907 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.303881    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:04.307621 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.307600    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-664tl"
Oct 30 03:57:04.484893 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.484852    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b67qk" event={"ID":"623e4b93-06e4-44f9-bd85-d9b663df4834","Type":"ContainerStarted","Data":"5bb9dedc69af52fd1e995ef87eb734be7b73e7f7bb8915c3f511589f0286525e"}
Oct 30 03:57:04.487680 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.487645    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sb794" event={"ID":"503b82e1-e56c-4c80-9d63-fa8e647c0991","Type":"ContainerStarted","Data":"711e1dd4a6e65055c9c3799042128281f66546e9e2a0b0b2c64307be1cdf68e6"}
Oct 30 03:57:04.489380 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.489355    5072 generic.go:356] "Generic (PLEG): container finished" podID="d41c0686-14d0-43c4-928e-e38b57021fb7" containerID="be3dd1a8d7dd162c3aa399e155ea9ae869da5a735b4514fc44940e03253c4c21" exitCode=0
Oct 30 03:57:04.489465 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.489445    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-trv7j" event={"ID":"d41c0686-14d0-43c4-928e-e38b57021fb7","Type":"ContainerDied","Data":"be3dd1a8d7dd162c3aa399e155ea9ae869da5a735b4514fc44940e03253c4c21"}
Oct 30 03:57:04.489809 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.489791    5072 scope.go:122] "RemoveContainer" containerID="be3dd1a8d7dd162c3aa399e155ea9ae869da5a735b4514fc44940e03253c4c21"
Oct 30 03:57:04.491700 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.491678    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz" event={"ID":"788f5892-601d-42cc-b597-528a091feac9","Type":"ContainerStarted","Data":"0ca4f2e601fd62e47af382d182dc3c6643edc1864fdb8024671b17b4a49154c3"}
Oct 30 03:57:04.491778 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.491718    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xccsz" event={"ID":"788f5892-601d-42cc-b597-528a091feac9","Type":"ContainerStarted","Data":"3bb100f6f90edc504a2a20b4b3ce6e3c63c61c9ee1fe986fdd88c14dd94a75bf"}
Oct 30 03:57:04.491847 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.491776    5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:57:04.533723 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.533693    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:04.749321 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.749285    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:04.929219 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.929191    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:57:04.987324 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:04.987241    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:05.169266 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.169231    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:05.169266 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.169265    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:05.169480 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.169299    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:57:05.169480 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.169331    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:05.169480 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.169362    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:05.169480 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:05.169391    5072 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:57:05.169480 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:05.169457    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls podName:0d2109e3-d762-4bf9-bf3a-d405148ec32c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:09.169440309 +0000 UTC m=+8.910863398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7x92g" (UID: "0d2109e3-d762-4bf9-bf3a-d405148ec32c") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:57:05.169480 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:05.169470    5072 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:57:05.169480 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:05.169475    5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:57:05.169730 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:05.169481    5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:57:05.169730 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:05.169493    5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-759fbb47f9-s2s2f: secret "image-registry-tls" not found
Oct 30 03:57:05.169730 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:05.169501    5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bbcfd588b-rlkpf: secret "image-registry-tls" not found
Oct 30 03:57:05.169730 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:05.169532    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert podName:01bc8bd9-f593-4acf-97c0-fc6c4beb4867 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:09.169514142 +0000 UTC m=+8.910937234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-htskn" (UID: "01bc8bd9-f593-4acf-97c0-fc6c4beb4867") : secret "networking-console-plugin-cert" not found
Oct 30 03:57:05.169730 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:05.169553    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls podName:343419d3-210d-4c12-86ff-c92f8f574b27 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:09.169540877 +0000 UTC m=+8.910963957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls") pod "image-registry-6bbcfd588b-rlkpf" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27") : secret "image-registry-tls" not found
Oct 30 03:57:05.169730 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:05.169568    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls podName:395d5cbd-c077-4578-b377-3034e42a2b94 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:09.169561181 +0000 UTC m=+8.910984256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls") pod "image-registry-759fbb47f9-s2s2f" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94") : secret "image-registry-tls" not found
Oct 30 03:57:05.171664 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.171635    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/730fcc37-b18e-46e3-aea5-a7fba60b7a8c-metrics-tls\") pod \"dns-default-xkq6m\" (UID: \"730fcc37-b18e-46e3-aea5-a7fba60b7a8c\") " pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:05.208975 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.208937    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:05.334945 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.334873    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:05.497577 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.497543    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b67qk" event={"ID":"623e4b93-06e4-44f9-bd85-d9b663df4834","Type":"ContainerStarted","Data":"81cd7f7c5204c4a0f96955c876239aa3949a5b330e060dbb741c103e46f2009e"}
Oct 30 03:57:05.498006 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.497605    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b67qk" event={"ID":"623e4b93-06e4-44f9-bd85-d9b663df4834","Type":"ContainerStarted","Data":"932a9c9579d58ea991e67afa1b2e5a34a2647a0f4ca8e42bce839cf67457d5e6"}
Oct 30 03:57:05.499539 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.499510    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-trv7j" event={"ID":"d41c0686-14d0-43c4-928e-e38b57021fb7","Type":"ContainerStarted","Data":"e12c76d44e05aac14a9f7a7da656e8d0e19ee8e7e2365753addea7b61b2ce1ff"}
Oct 30 03:57:05.545381 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.545349    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:05.631732 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.631710    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xkq6m"]
Oct 30 03:57:05.633616 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:05.633589    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730fcc37_b18e_46e3_aea5_a7fba60b7a8c.slice/crio-64adacca002bf7bd9d9c1cc079f9a9fc33fb54316c8904998b7bb53cdca7928e WatchSource:0}: Error finding container 64adacca002bf7bd9d9c1cc079f9a9fc33fb54316c8904998b7bb53cdca7928e: Status 404 returned error can't find the container with id 64adacca002bf7bd9d9c1cc079f9a9fc33fb54316c8904998b7bb53cdca7928e
Oct 30 03:57:05.754928 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.754891    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:05.889973 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:05.889903    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:06.038663 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:06.038627    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:06.348874 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:06.348837    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:06.505131 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:06.505101    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xkq6m" event={"ID":"730fcc37-b18e-46e3-aea5-a7fba60b7a8c","Type":"ContainerStarted","Data":"64adacca002bf7bd9d9c1cc079f9a9fc33fb54316c8904998b7bb53cdca7928e"}
Oct 30 03:57:07.168028 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:07.167998    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:07.391349 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:07.391263    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:07.510971 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:07.510937    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xkq6m" event={"ID":"730fcc37-b18e-46e3-aea5-a7fba60b7a8c","Type":"ContainerStarted","Data":"aacd3010c021f93f28dcfcdafeeadc419acbd244e0061fd07a66c6c3c4d2f37b"}
Oct 30 03:57:07.511349 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:07.510989    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xkq6m" event={"ID":"730fcc37-b18e-46e3-aea5-a7fba60b7a8c","Type":"ContainerStarted","Data":"86620b864eb3048023637c6bb7530e2d002e3634fbe308bde5b5e2bee705b803"}
Oct 30 03:57:07.511349 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:07.511068    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:07.751742 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:07.751708    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:07.962844 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:07.962808    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:08.554857 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:08.554822    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:08.840792 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:08.840710    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:08.840930 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:08.840899    5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:57:08.868689 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:08.868658    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:09.079188 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.079152    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:09.208997 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.208972    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:09.209144 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.209016    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:57:09.209144 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.209039    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:09.209144 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.209064    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:09.211662 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.211633    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"image-registry-759fbb47f9-s2s2f\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") " pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:09.211785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.211676    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01bc8bd9-f593-4acf-97c0-fc6c4beb4867-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-htskn\" (UID: \"01bc8bd9-f593-4acf-97c0-fc6c4beb4867\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:57:09.211785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.211678    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"image-registry-6bbcfd588b-rlkpf\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") " pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:09.211785 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.211678    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d2109e3-d762-4bf9-bf3a-d405148ec32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7x92g\" (UID: \"0d2109e3-d762-4bf9-bf3a-d405148ec32c\") " pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:09.219454 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.219011    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:09.223500 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.223478    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn"
Oct 30 03:57:09.236164 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.236128    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7x92g"
Oct 30 03:57:09.236394 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.236343    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:09.294483 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.294323    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:09.510541 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.510450    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:09.549538 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.549512    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-htskn"]
Oct 30 03:57:09.551176 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:09.551140    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01bc8bd9_f593_4acf_97c0_fc6c4beb4867.slice/crio-07a52c744e1ec6f09fea80d1dac9c831dd0c8a73023a3c95b1cabedbfa8ea51d WatchSource:0}: Error finding container 07a52c744e1ec6f09fea80d1dac9c831dd0c8a73023a3c95b1cabedbfa8ea51d: Status 404 returned error can't find the container with id 07a52c744e1ec6f09fea80d1dac9c831dd0c8a73023a3c95b1cabedbfa8ea51d
Oct 30 03:57:09.553608 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.553589    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-759fbb47f9-s2s2f"]
Oct 30 03:57:09.554772 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:09.554749    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395d5cbd_c077_4578_b377_3034e42a2b94.slice/crio-9d33f03496825199ccb3483b58630616822b3e3acdcd451cd3677f63888a8ee4 WatchSource:0}: Error finding container 9d33f03496825199ccb3483b58630616822b3e3acdcd451cd3677f63888a8ee4: Status 404 returned error can't find the container with id 9d33f03496825199ccb3483b58630616822b3e3acdcd451cd3677f63888a8ee4
Oct 30 03:57:09.562429 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.562405    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7x92g"]
Oct 30 03:57:09.564430 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:09.564411    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2109e3_d762_4bf9_bf3a_d405148ec32c.slice/crio-116b2b4ecf0ad12dd7ebb78f102b5058ff0a08169f1e43694dbb504a865260f4 WatchSource:0}: Error finding container 116b2b4ecf0ad12dd7ebb78f102b5058ff0a08169f1e43694dbb504a865260f4: Status 404 returned error can't find the container with id 116b2b4ecf0ad12dd7ebb78f102b5058ff0a08169f1e43694dbb504a865260f4
Oct 30 03:57:09.569401 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:09.569380    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bbcfd588b-rlkpf"]
Oct 30 03:57:09.570733 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:09.570706    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343419d3_210d_4c12_86ff_c92f8f574b27.slice/crio-7e5d7902ec58915cda31f78ed7be07838036933f99c510215cf82465dea8cfe0 WatchSource:0}: Error finding container 7e5d7902ec58915cda31f78ed7be07838036933f99c510215cf82465dea8cfe0: Status 404 returned error can't find the container with id 7e5d7902ec58915cda31f78ed7be07838036933f99c510215cf82465dea8cfe0
Oct 30 03:57:10.524487 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:10.524453    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f" event={"ID":"395d5cbd-c077-4578-b377-3034e42a2b94","Type":"ContainerStarted","Data":"833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd"}
Oct 30 03:57:10.524694 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:10.524512    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f" event={"ID":"395d5cbd-c077-4578-b377-3034e42a2b94","Type":"ContainerStarted","Data":"9d33f03496825199ccb3483b58630616822b3e3acdcd451cd3677f63888a8ee4"}
Oct 30 03:57:10.526450 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:10.526417    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7x92g" event={"ID":"0d2109e3-d762-4bf9-bf3a-d405148ec32c","Type":"ContainerStarted","Data":"fa77abd079630f825a27a1de92fbc165eaf74c258dd6ad3668b7749cb5b9b40e"}
Oct 30 03:57:10.526572 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:10.526467    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7x92g" event={"ID":"0d2109e3-d762-4bf9-bf3a-d405148ec32c","Type":"ContainerStarted","Data":"b602e5f479c4071c3a7db3d56b7e0c758247a7d9ab0aa1ebc77f6a1e70711906"}
Oct 30 03:57:10.526572 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:10.526485    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7x92g" event={"ID":"0d2109e3-d762-4bf9-bf3a-d405148ec32c","Type":"ContainerStarted","Data":"116b2b4ecf0ad12dd7ebb78f102b5058ff0a08169f1e43694dbb504a865260f4"}
Oct 30 03:57:10.527550 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:10.527528    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn" event={"ID":"01bc8bd9-f593-4acf-97c0-fc6c4beb4867","Type":"ContainerStarted","Data":"07a52c744e1ec6f09fea80d1dac9c831dd0c8a73023a3c95b1cabedbfa8ea51d"}
Oct 30 03:57:10.529235 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:10.529201    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf" event={"ID":"343419d3-210d-4c12-86ff-c92f8f574b27","Type":"ContainerStarted","Data":"4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3"}
Oct 30 03:57:10.529327 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:10.529243    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf" event={"ID":"343419d3-210d-4c12-86ff-c92f8f574b27","Type":"ContainerStarted","Data":"7e5d7902ec58915cda31f78ed7be07838036933f99c510215cf82465dea8cfe0"}
Oct 30 03:57:11.538959 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:11.538868    5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:57:11.538959 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:11.538877    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-htskn" event={"ID":"01bc8bd9-f593-4acf-97c0-fc6c4beb4867","Type":"ContainerStarted","Data":"7789e780d24e44944825847e6c563cf5d8e175d0bf6ef0c665b2e4b301761408"}
Oct 30 03:57:11.538959 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:11.538903    5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:57:11.628575 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:11.628545    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:11.628710 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:11.628684    5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:57:11.630495 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:11.630476    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-84bd8c5fd7-885hp"
Oct 30 03:57:12.545231 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:12.545189    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7x92g" event={"ID":"0d2109e3-d762-4bf9-bf3a-d405148ec32c","Type":"ContainerStarted","Data":"510f510cb093fb962c4564b9abcf6cdcfdb19410007af2b8c82106e34be69d79"}
Oct 30 03:57:13.875377 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:13.875343    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:14.010906 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:14.010879    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:14.089521 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:14.089484    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:14.999827 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:14.999793    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-crtnn-gt78p"]
Oct 30 03:57:15.011128 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.011070    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.030170 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.030138    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-crtnn-gt78p"]
Oct 30 03:57:15.030305 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.030208    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-759fbb47f9-s2s2f"]
Oct 30 03:57:15.030498 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.030477    5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:57:15.043499 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.043469    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85c4fcd595-5xn9h"]
Oct 30 03:57:15.047262 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.047227    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-j55cl"]
Oct 30 03:57:15.047483 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.047454    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.051050 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.050976    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j55cl"
Oct 30 03:57:15.063002 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.062973    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c4fcd595-5xn9h"]
Oct 30 03:57:15.068655 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.068625    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gmjld\""
Oct 30 03:57:15.068786 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.068701    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 03:57:15.072142 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.072119    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 03:57:15.072254 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.072190    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 03:57:15.072799 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.072532    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 03:57:15.072799 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.072620    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 03:57:15.074446 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.074423    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 03:57:15.074540 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.074477    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 03:57:15.095381 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.095354    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j55cl"]
Oct 30 03:57:15.136619 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.132816    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv"]
Oct 30 03:57:15.144815 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.144791    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6878899c59-ksp2c"]
Oct 30 03:57:15.144932 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.144900    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv"
Oct 30 03:57:15.148106 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.148068    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.168753 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.168726    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv"]
Oct 30 03:57:15.169006 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.168981    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-service-ca\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.169098 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169025    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bb6afde-1ba7-4045-841d-528387cf7f9d-tmp\") pod \"downloads-6dd789c8b7-j55cl\" (UID: \"5bb6afde-1ba7-4045-841d-528387cf7f9d\") " pod="openshift-console/downloads-6dd789c8b7-j55cl"
Oct 30 03:57:15.169098 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169052    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63ac0921-404c-493d-9aec-02c93599b494-tmp\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169109    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-console-config\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169302    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/247ce32a-598a-4d86-9892-12af06a4c753-archives-path\") pod \"periodic-gathering-crtnn-gt78p\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") " pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169367    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-oauth-serving-cert\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169387    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-service-ca-bundle\") pod \"periodic-gathering-crtnn-gt78p\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") " pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169491    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-oauth-config\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169559    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-serving-cert\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169592    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-trusted-ca-bundle\") pod \"periodic-gathering-crtnn-gt78p\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") " pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169685    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zspg8\" (UniqueName: \"kubernetes.io/projected/63ac0921-404c-493d-9aec-02c93599b494-kube-api-access-zspg8\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169724    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljf6f\" (UniqueName: \"kubernetes.io/projected/247ce32a-598a-4d86-9892-12af06a4c753-kube-api-access-ljf6f\") pod \"periodic-gathering-crtnn-gt78p\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") " pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169753    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd6rl\" (UniqueName: \"kubernetes.io/projected/5bb6afde-1ba7-4045-841d-528387cf7f9d-kube-api-access-wd6rl\") pod \"downloads-6dd789c8b7-j55cl\" (UID: \"5bb6afde-1ba7-4045-841d-528387cf7f9d\") " pod="openshift-console/downloads-6dd789c8b7-j55cl"
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169893    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.169952    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-mmgrt\""
Oct 30 03:57:15.171172 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.170002    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-smxbq\""
Oct 30 03:57:15.203765 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.203737    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6878899c59-ksp2c"]
Oct 30 03:57:15.270964 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.270892    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-serving-cert\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.270964 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.270937    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-trusted-ca-bundle\") pod \"periodic-gathering-crtnn-gt78p\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") " pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.271227 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.270976    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zspg8\" (UniqueName: \"kubernetes.io/projected/63ac0921-404c-493d-9aec-02c93599b494-kube-api-access-zspg8\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.271227 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271002    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljf6f\" (UniqueName: \"kubernetes.io/projected/247ce32a-598a-4d86-9892-12af06a4c753-kube-api-access-ljf6f\") pod \"periodic-gathering-crtnn-gt78p\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") " pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.271227 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271030    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd6rl\" (UniqueName: \"kubernetes.io/projected/5bb6afde-1ba7-4045-841d-528387cf7f9d-kube-api-access-wd6rl\") pod \"downloads-6dd789c8b7-j55cl\" (UID: \"5bb6afde-1ba7-4045-841d-528387cf7f9d\") " pod="openshift-console/downloads-6dd789c8b7-j55cl"
Oct 30 03:57:15.271227 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271153    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8shm\" (UniqueName: \"kubernetes.io/projected/243e61df-e18d-402e-bef1-f2713f198a08-kube-api-access-t8shm\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.271367 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271221    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/243e61df-e18d-402e-bef1-f2713f198a08-image-registry-private-configuration\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.271367 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271314    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-service-ca\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.271367 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271350    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bb6afde-1ba7-4045-841d-528387cf7f9d-tmp\") pod \"downloads-6dd789c8b7-j55cl\" (UID: \"5bb6afde-1ba7-4045-841d-528387cf7f9d\") " pod="openshift-console/downloads-6dd789c8b7-j55cl"
Oct 30 03:57:15.271480 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271382    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63ac0921-404c-493d-9aec-02c93599b494-tmp\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.271480 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271410    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/243e61df-e18d-402e-bef1-f2713f198a08-bound-sa-token\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.271480 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271442    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4e25fafd-0ea2-41d2-a9cf-2a8b1c9fa6b6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lwsgv\" (UID: \"4e25fafd-0ea2-41d2-a9cf-2a8b1c9fa6b6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv"
Oct 30 03:57:15.271480 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271472    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-console-config\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.271668 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271654    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/243e61df-e18d-402e-bef1-f2713f198a08-trusted-ca\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.271728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271687    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/247ce32a-598a-4d86-9892-12af06a4c753-archives-path\") pod \"periodic-gathering-crtnn-gt78p\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") " pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.271728 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271717    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bb6afde-1ba7-4045-841d-528387cf7f9d-tmp\") pod \"downloads-6dd789c8b7-j55cl\" (UID: \"5bb6afde-1ba7-4045-841d-528387cf7f9d\") " pod="openshift-console/downloads-6dd789c8b7-j55cl"
Oct 30 03:57:15.271830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271743    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/243e61df-e18d-402e-bef1-f2713f198a08-registry-certificates\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.271830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271753    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63ac0921-404c-493d-9aec-02c93599b494-tmp\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.271830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271808    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-oauth-serving-cert\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.271973 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271836    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-service-ca-bundle\") pod \"periodic-gathering-crtnn-gt78p\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") " pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.271973 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271877    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/243e61df-e18d-402e-bef1-f2713f198a08-installation-pull-secrets\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.271973 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271916    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/243e61df-e18d-402e-bef1-f2713f198a08-registry-tls\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.271973 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271960    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-oauth-config\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.272190 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.271991    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/243e61df-e18d-402e-bef1-f2713f198a08-ca-trust-extracted\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.272190 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.272062    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-service-ca\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.272190 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.272171    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/247ce32a-598a-4d86-9892-12af06a4c753-archives-path\") pod \"periodic-gathering-crtnn-gt78p\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") " pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.272332 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.272288    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-console-config\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.272487 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.272470    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-service-ca-bundle\") pod \"periodic-gathering-crtnn-gt78p\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") " pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.272692 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.272672    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-oauth-serving-cert\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.272788 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.272774    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-trusted-ca-bundle\") pod \"periodic-gathering-crtnn-gt78p\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") " pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.274020 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.273996    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-serving-cert\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.275109 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.275043    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-oauth-config\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.306723 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.306689    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zspg8\" (UniqueName: \"kubernetes.io/projected/63ac0921-404c-493d-9aec-02c93599b494-kube-api-access-zspg8\") pod \"console-85c4fcd595-5xn9h\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") " pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.309354 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.309324    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:15.309674 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.309658    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljf6f\" (UniqueName: \"kubernetes.io/projected/247ce32a-598a-4d86-9892-12af06a4c753-kube-api-access-ljf6f\") pod \"periodic-gathering-crtnn-gt78p\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") " pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.309815 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.309794    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd6rl\" (UniqueName: \"kubernetes.io/projected/5bb6afde-1ba7-4045-841d-528387cf7f9d-kube-api-access-wd6rl\") pod \"downloads-6dd789c8b7-j55cl\" (UID: \"5bb6afde-1ba7-4045-841d-528387cf7f9d\") " pod="openshift-console/downloads-6dd789c8b7-j55cl"
Oct 30 03:57:15.337881 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.337638    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:57:15.369312 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.369280    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:15.372647 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.372620    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/243e61df-e18d-402e-bef1-f2713f198a08-trusted-ca\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.372770 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.372677    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/243e61df-e18d-402e-bef1-f2713f198a08-registry-certificates\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.372770 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.372716    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/243e61df-e18d-402e-bef1-f2713f198a08-installation-pull-secrets\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.372770 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.372741    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/243e61df-e18d-402e-bef1-f2713f198a08-registry-tls\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.372924 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.372782    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/243e61df-e18d-402e-bef1-f2713f198a08-ca-trust-extracted\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.372924 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.372820    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8shm\" (UniqueName: \"kubernetes.io/projected/243e61df-e18d-402e-bef1-f2713f198a08-kube-api-access-t8shm\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.372924 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.372856    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/243e61df-e18d-402e-bef1-f2713f198a08-image-registry-private-configuration\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.372924 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.372910    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/243e61df-e18d-402e-bef1-f2713f198a08-bound-sa-token\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.373116 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.372940    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4e25fafd-0ea2-41d2-a9cf-2a8b1c9fa6b6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lwsgv\" (UID: \"4e25fafd-0ea2-41d2-a9cf-2a8b1c9fa6b6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv"
Oct 30 03:57:15.374809 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.374784    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/243e61df-e18d-402e-bef1-f2713f198a08-ca-trust-extracted\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.374954 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.374927    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/243e61df-e18d-402e-bef1-f2713f198a08-registry-certificates\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.374954 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.374944    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/243e61df-e18d-402e-bef1-f2713f198a08-trusted-ca\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.376501 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.376481    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4e25fafd-0ea2-41d2-a9cf-2a8b1c9fa6b6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lwsgv\" (UID: \"4e25fafd-0ea2-41d2-a9cf-2a8b1c9fa6b6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv"
Oct 30 03:57:15.377389 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.377374    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/243e61df-e18d-402e-bef1-f2713f198a08-registry-tls\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.377448 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.377434    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/243e61df-e18d-402e-bef1-f2713f198a08-installation-pull-secrets\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.379217 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.379193    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/243e61df-e18d-402e-bef1-f2713f198a08-image-registry-private-configuration\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.391775 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.391338    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j55cl"
Oct 30 03:57:15.409840 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.409814    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/243e61df-e18d-402e-bef1-f2713f198a08-bound-sa-token\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.411960 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.411939    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8shm\" (UniqueName: \"kubernetes.io/projected/243e61df-e18d-402e-bef1-f2713f198a08-kube-api-access-t8shm\") pod \"image-registry-6878899c59-ksp2c\" (UID: \"243e61df-e18d-402e-bef1-f2713f198a08\") " pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.463700 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.463428    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:15.466235 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.464032    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv"
Oct 30 03:57:15.568827 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.568694    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:15.678934 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.678904    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-crtnn-gt78p"]
Oct 30 03:57:15.682491 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:15.682467    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247ce32a_598a_4d86_9892_12af06a4c753.slice/crio-42a9081f3bd6a41c13de4d25ece9820f3aa80adc302aae6cfca37849fb556544 WatchSource:0}: Error finding container 42a9081f3bd6a41c13de4d25ece9820f3aa80adc302aae6cfca37849fb556544: Status 404 returned error can't find the container with id 42a9081f3bd6a41c13de4d25ece9820f3aa80adc302aae6cfca37849fb556544
Oct 30 03:57:15.742133 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.742109    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c4fcd595-5xn9h"]
Oct 30 03:57:15.749777 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:15.749685    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ac0921_404c_493d_9aec_02c93599b494.slice/crio-f798d77e62b0f336d1ec61dab05c1fe54e84fc76bae7edf583ab5acb2e8fad68 WatchSource:0}: Error finding container f798d77e62b0f336d1ec61dab05c1fe54e84fc76bae7edf583ab5acb2e8fad68: Status 404 returned error can't find the container with id f798d77e62b0f336d1ec61dab05c1fe54e84fc76bae7edf583ab5acb2e8fad68
Oct 30 03:57:15.750670 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.750639    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j55cl"]
Oct 30 03:57:15.751871 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:15.751845    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb6afde_1ba7_4045_841d_528387cf7f9d.slice/crio-65bc2c4541ee5bb4c951909179d8db96c21d17de4d5cdf3a38f5fb9e8e8bd3ca WatchSource:0}: Error finding container 65bc2c4541ee5bb4c951909179d8db96c21d17de4d5cdf3a38f5fb9e8e8bd3ca: Status 404 returned error can't find the container with id 65bc2c4541ee5bb4c951909179d8db96c21d17de4d5cdf3a38f5fb9e8e8bd3ca
Oct 30 03:57:15.827453 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.827414    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv"]
Oct 30 03:57:15.844952 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:15.844927    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6878899c59-ksp2c"]
Oct 30 03:57:15.848334 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:15.848303    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243e61df_e18d_402e_bef1_f2713f198a08.slice/crio-c9fd852bfccc13c2d0584a297bb83212d514acfe9b3ab1fac368d17d5199497a WatchSource:0}: Error finding container c9fd852bfccc13c2d0584a297bb83212d514acfe9b3ab1fac368d17d5199497a: Status 404 returned error can't find the container with id c9fd852bfccc13c2d0584a297bb83212d514acfe9b3ab1fac368d17d5199497a
Oct 30 03:57:16.577990 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:16.577895    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6878899c59-ksp2c" event={"ID":"243e61df-e18d-402e-bef1-f2713f198a08","Type":"ContainerStarted","Data":"68cff689653a0530fe2a0b8b737ec5ab32d6ac36142149ef70e578340673ff0f"}
Oct 30 03:57:16.577990 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:16.577955    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6878899c59-ksp2c" event={"ID":"243e61df-e18d-402e-bef1-f2713f198a08","Type":"ContainerStarted","Data":"c9fd852bfccc13c2d0584a297bb83212d514acfe9b3ab1fac368d17d5199497a"}
Oct 30 03:57:16.578536 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:16.578009    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:16.580638 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:16.580572    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c4fcd595-5xn9h" event={"ID":"63ac0921-404c-493d-9aec-02c93599b494","Type":"ContainerStarted","Data":"f798d77e62b0f336d1ec61dab05c1fe54e84fc76bae7edf583ab5acb2e8fad68"}
Oct 30 03:57:16.582960 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:16.582933    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j55cl" event={"ID":"5bb6afde-1ba7-4045-841d-528387cf7f9d","Type":"ContainerStarted","Data":"65bc2c4541ee5bb4c951909179d8db96c21d17de4d5cdf3a38f5fb9e8e8bd3ca"}
Oct 30 03:57:16.586644 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:16.586620    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-crtnn-gt78p" event={"ID":"247ce32a-598a-4d86-9892-12af06a4c753","Type":"ContainerStarted","Data":"c54ea84650da135a603d64055f96b6713691f66bf810ad7c771849269b304d18"}
Oct 30 03:57:16.586725 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:16.586662    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-crtnn-gt78p" event={"ID":"247ce32a-598a-4d86-9892-12af06a4c753","Type":"ContainerStarted","Data":"42a9081f3bd6a41c13de4d25ece9820f3aa80adc302aae6cfca37849fb556544"}
Oct 30 03:57:16.595457 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:16.594173    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv" event={"ID":"4e25fafd-0ea2-41d2-a9cf-2a8b1c9fa6b6","Type":"ContainerStarted","Data":"3697bca7c4302b393eb33ae7b183930becd6d7a1f4fd635f3e47da7105525239"}
Oct 30 03:57:16.625188 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:16.624778    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6878899c59-ksp2c" podStartSLOduration=1.624761179 podStartE2EDuration="1.624761179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:16.624409656 +0000 UTC m=+16.365832753" watchObservedRunningTime="2026-10-30 03:57:16.624761179 +0000 UTC m=+16.366184275"
Oct 30 03:57:16.667860 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:16.667764    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-crtnn-gt78p" podStartSLOduration=2.667742449 podStartE2EDuration="2.667742449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:16.66606724 +0000 UTC m=+16.407490344" watchObservedRunningTime="2026-10-30 03:57:16.667742449 +0000 UTC m=+16.409165556"
Oct 30 03:57:16.790474 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:16.790344    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xkq6m_730fcc37-b18e-46e3-aea5-a7fba60b7a8c/dns/0.log"
Oct 30 03:57:16.982253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:16.982227    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xkq6m_730fcc37-b18e-46e3-aea5-a7fba60b7a8c/kube-rbac-proxy/0.log"
Oct 30 03:57:17.129551 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.129465    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:17.184700 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.184467    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6njd4_b555cf82-5423-4c47-a004-ae399654683b/dns-node-resolver/0.log"
Oct 30 03:57:17.298779 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.298704    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:17.383091 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.383058    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6878899c59-ksp2c_243e61df-e18d-402e-bef1-f2713f198a08/registry/0.log"
Oct 30 03:57:17.517139 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.516855    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xkq6m"
Oct 30 03:57:17.526909 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.526866    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:17.587814 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.583211    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6bbcfd588b-rlkpf_343419d3-210d-4c12-86ff-c92f8f574b27/registry/0.log"
Oct 30 03:57:17.604498 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.604248    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv" event={"ID":"4e25fafd-0ea2-41d2-a9cf-2a8b1c9fa6b6","Type":"ContainerStarted","Data":"51b8273470fd5ac9c1ee55e3334bcf954b1b3b085793b103bd84577f00906e59"}
Oct 30 03:57:17.605364 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.605339    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv"
Oct 30 03:57:17.612109 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.610836    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv"
Oct 30 03:57:17.651791 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.651735    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwsgv" podStartSLOduration=1.114001213 podStartE2EDuration="2.651722464s" totalImagesPullingTime="1.537721251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:17.651489455 +0000 UTC m=+17.392912555" watchObservedRunningTime="2026-10-30 03:57:17.651722464 +0000 UTC m=+17.393145560"
Oct 30 03:57:17.779924 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.779891    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-759fbb47f9-s2s2f_395d5cbd-c077-4578-b377-3034e42a2b94/registry/0.log"
Oct 30 03:57:17.980747 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:17.980720    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mc77t_0384e218-1937-4099-8566-67775cda0521/node-ca/0.log"
Oct 30 03:57:18.083513 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.083471    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:18.085051 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.085024    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"]
Oct 30 03:57:18.099965 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.099497    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.112857 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.112826    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"]
Oct 30 03:57:18.124373 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.124293    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 03:57:18.124373 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.124338    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 03:57:18.124762 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.124690    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 03:57:18.124762 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.124747    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l4l6w\""
Oct 30 03:57:18.179826 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.179792    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84bd8c5fd7-885hp_b752c121-5e44-4df2-9f17-97f0c899516b/router/0.log"
Oct 30 03:57:18.218307 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.218051    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/271ee094-c03d-4a43-b81d-d11b7262ff36-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6vzvm\" (UID: \"271ee094-c03d-4a43-b81d-d11b7262ff36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.218307 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.218137    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz7n8\" (UniqueName: \"kubernetes.io/projected/271ee094-c03d-4a43-b81d-d11b7262ff36-kube-api-access-kz7n8\") pod \"prometheus-operator-9cd89b74c-6vzvm\" (UID: \"271ee094-c03d-4a43-b81d-d11b7262ff36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.218307 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.218166    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/271ee094-c03d-4a43-b81d-d11b7262ff36-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6vzvm\" (UID: \"271ee094-c03d-4a43-b81d-d11b7262ff36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.218307 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.218199    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/271ee094-c03d-4a43-b81d-d11b7262ff36-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6vzvm\" (UID: \"271ee094-c03d-4a43-b81d-d11b7262ff36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.305532 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.305405    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:18.319509 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.319478    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/271ee094-c03d-4a43-b81d-d11b7262ff36-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6vzvm\" (UID: \"271ee094-c03d-4a43-b81d-d11b7262ff36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.319830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.319543    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7n8\" (UniqueName: \"kubernetes.io/projected/271ee094-c03d-4a43-b81d-d11b7262ff36-kube-api-access-kz7n8\") pod \"prometheus-operator-9cd89b74c-6vzvm\" (UID: \"271ee094-c03d-4a43-b81d-d11b7262ff36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.319830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.319586    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/271ee094-c03d-4a43-b81d-d11b7262ff36-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6vzvm\" (UID: \"271ee094-c03d-4a43-b81d-d11b7262ff36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.319830 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.319651    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/271ee094-c03d-4a43-b81d-d11b7262ff36-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6vzvm\" (UID: \"271ee094-c03d-4a43-b81d-d11b7262ff36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.320350 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.320324    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/271ee094-c03d-4a43-b81d-d11b7262ff36-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6vzvm\" (UID: \"271ee094-c03d-4a43-b81d-d11b7262ff36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.322912 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.322868    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/271ee094-c03d-4a43-b81d-d11b7262ff36-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6vzvm\" (UID: \"271ee094-c03d-4a43-b81d-d11b7262ff36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.332842 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.332795    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/271ee094-c03d-4a43-b81d-d11b7262ff36-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6vzvm\" (UID: \"271ee094-c03d-4a43-b81d-d11b7262ff36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.348715 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.348691    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz7n8\" (UniqueName: \"kubernetes.io/projected/271ee094-c03d-4a43-b81d-d11b7262ff36-kube-api-access-kz7n8\") pod \"prometheus-operator-9cd89b74c-6vzvm\" (UID: \"271ee094-c03d-4a43-b81d-d11b7262ff36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.379043 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.379015    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bh464_89f2993a-ed5d-488e-832d-bcba8ddd9706/serve-healthcheck-canary/0.log"
Oct 30 03:57:18.430226 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.430192    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"
Oct 30 03:57:18.524412 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.524324    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:18.742740 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.742702    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm"]
Oct 30 03:57:18.750013 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:18.749984    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271ee094_c03d_4a43_b81d_d11b7262ff36.slice/crio-170e6565125cd44fba062dc85737eba62f6d37e43f44859aae0d59a9a6f5e448 WatchSource:0}: Error finding container 170e6565125cd44fba062dc85737eba62f6d37e43f44859aae0d59a9a6f5e448: Status 404 returned error can't find the container with id 170e6565125cd44fba062dc85737eba62f6d37e43f44859aae0d59a9a6f5e448
Oct 30 03:57:18.935355 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.935321    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8565dfb875-f75g8"]
Oct 30 03:57:18.944156 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.944128    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:18.959447 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:18.959384    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8565dfb875-f75g8"]
Oct 30 03:57:19.030573 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.030501    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 03:57:19.126974 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.126939    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9ppn\" (UniqueName: \"kubernetes.io/projected/716a3fab-e121-4291-8ef5-02a47719ccbb-kube-api-access-w9ppn\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.127164 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.127009    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-oauth-config\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.127164 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.127038    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-trusted-ca-bundle\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.127164 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.127106    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-service-ca\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.127164 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.127129    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/716a3fab-e121-4291-8ef5-02a47719ccbb-tmp\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.127372 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.127209    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-oauth-serving-cert\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.127372 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.127264    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-serving-cert\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.127372 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.127313    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-console-config\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.228554 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.228517    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-oauth-serving-cert\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.228732 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.228570    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-serving-cert\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.228732 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.228615    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-console-config\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.228732 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.228649    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9ppn\" (UniqueName: \"kubernetes.io/projected/716a3fab-e121-4291-8ef5-02a47719ccbb-kube-api-access-w9ppn\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.228732 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.228698    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-oauth-config\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.228732 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.228721    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-trusted-ca-bundle\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.228895 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.228755    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-service-ca\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.228895 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.228771    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/716a3fab-e121-4291-8ef5-02a47719ccbb-tmp\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.229511 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.229094    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/716a3fab-e121-4291-8ef5-02a47719ccbb-tmp\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.229511 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.229358    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-oauth-serving-cert\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.229994 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.229942    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-console-config\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.230567 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.230540    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-service-ca\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.231268 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.231246    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-trusted-ca-bundle\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.231998 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.231974    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-oauth-config\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.251272 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.251244    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-serving-cert\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.254874 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.254829    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9ppn\" (UniqueName: \"kubernetes.io/projected/716a3fab-e121-4291-8ef5-02a47719ccbb-kube-api-access-w9ppn\") pod \"console-8565dfb875-f75g8\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") " pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.262664 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.262642    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:19.578566 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.578451    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8565dfb875-f75g8"]
Oct 30 03:57:19.581720 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:19.581680    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716a3fab_e121_4291_8ef5_02a47719ccbb.slice/crio-9976bcf354c95de1797f4f369c9d93a396826849657f72b143371869fcf7f81e WatchSource:0}: Error finding container 9976bcf354c95de1797f4f369c9d93a396826849657f72b143371869fcf7f81e: Status 404 returned error can't find the container with id 9976bcf354c95de1797f4f369c9d93a396826849657f72b143371869fcf7f81e
Oct 30 03:57:19.618673 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.618639    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm" event={"ID":"271ee094-c03d-4a43-b81d-d11b7262ff36","Type":"ContainerStarted","Data":"170e6565125cd44fba062dc85737eba62f6d37e43f44859aae0d59a9a6f5e448"}
Oct 30 03:57:19.622439 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.622411    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c4fcd595-5xn9h" event={"ID":"63ac0921-404c-493d-9aec-02c93599b494","Type":"ContainerStarted","Data":"fab5b5292926f9e51a6d4f52ff40d8f346edb5f1dd7e29143493feaa7fa09802"}
Oct 30 03:57:19.624588 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.624536    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8565dfb875-f75g8" event={"ID":"716a3fab-e121-4291-8ef5-02a47719ccbb","Type":"ContainerStarted","Data":"9976bcf354c95de1797f4f369c9d93a396826849657f72b143371869fcf7f81e"}
Oct 30 03:57:19.668431 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:19.666770    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-85c4fcd595-5xn9h" podStartSLOduration=1.065822623 podStartE2EDuration="4.666749262s" totalImagesPullingTime="3.600926639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:19.666549449 +0000 UTC m=+19.407972558" watchObservedRunningTime="2026-10-30 03:57:19.666749262 +0000 UTC m=+19.408172359"
Oct 30 03:57:20.631795 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:20.631756    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8565dfb875-f75g8" event={"ID":"716a3fab-e121-4291-8ef5-02a47719ccbb","Type":"ContainerStarted","Data":"81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c"}
Oct 30 03:57:20.634066 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:20.634033    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm" event={"ID":"271ee094-c03d-4a43-b81d-d11b7262ff36","Type":"ContainerStarted","Data":"3e1c152a83be2f16ddf6b694e5b3db32149d956beddfec24719d58c7595a7781"}
Oct 30 03:57:20.634230 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:20.634103    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm" event={"ID":"271ee094-c03d-4a43-b81d-d11b7262ff36","Type":"ContainerStarted","Data":"89407ed50616cb9ad4ec533041e7183a288a365b3e7da109bc46fdbb602102a5"}
Oct 30 03:57:20.676298 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:20.676250    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8565dfb875-f75g8" podStartSLOduration=2.676230463 podStartE2EDuration="2.676230463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:20.675600037 +0000 UTC m=+20.417023133" watchObservedRunningTime="2026-10-30 03:57:20.676230463 +0000 UTC m=+20.417653560"
Oct 30 03:57:20.718321 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:20.718270    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vzvm" podStartSLOduration=1.398739943 podStartE2EDuration="2.718252643s" totalImagesPullingTime="1.3195127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:20.71749705 +0000 UTC m=+20.458920155" watchObservedRunningTime="2026-10-30 03:57:20.718252643 +0000 UTC m=+20.459675739"
Oct 30 03:57:21.170633 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:21.170591    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:21.407355 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:21.407315    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:21.859988 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:21.859958    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d7nsr"
Oct 30 03:57:22.558131 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.558097    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:22.720118 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.718418    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"]
Oct 30 03:57:22.764458 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.764427    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"]
Oct 30 03:57:22.764640 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.764565    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:22.787284 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.787254    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"]
Oct 30 03:57:22.798218 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.798181    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 03:57:22.798569 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.798527    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 03:57:22.798569 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.798537    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:22.798750 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.798603    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fsgqg\""
Oct 30 03:57:22.822799 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.822285    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h78rv"]
Oct 30 03:57:22.823148 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.823109    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:22.846753 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.846735    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"]
Oct 30 03:57:22.846879 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.846856    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:22.858133 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.858115    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 03:57:22.866679 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.866658    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebb63468-4a79-401b-ada4-5b2e1ac38197-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wmrjs\" (UID: \"ebb63468-4a79-401b-ada4-5b2e1ac38197\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:22.866988 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.866695    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebb63468-4a79-401b-ada4-5b2e1ac38197-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wmrjs\" (UID: \"ebb63468-4a79-401b-ada4-5b2e1ac38197\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:22.866988 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.866728    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebb63468-4a79-401b-ada4-5b2e1ac38197-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wmrjs\" (UID: \"ebb63468-4a79-401b-ada4-5b2e1ac38197\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:22.866988 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.866811    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wllck\" (UniqueName: \"kubernetes.io/projected/ebb63468-4a79-401b-ada4-5b2e1ac38197-kube-api-access-wllck\") pod \"openshift-state-metrics-7bd7489957-wmrjs\" (UID: \"ebb63468-4a79-401b-ada4-5b2e1ac38197\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:22.888805 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.888782    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 03:57:22.889009 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.888871    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 03:57:22.889009 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.888893    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 03:57:22.889009 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.888939    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 03:57:22.889009 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.888954    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hlsvx\""
Oct 30 03:57:22.889009 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.889004    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8s8mj\""
Oct 30 03:57:22.894829 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.894810    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 03:57:22.968464 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968402    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c61d535d-a6ff-4a10-84af-9c697e073ddb-metrics-client-ca\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:22.968603 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968494    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a90b92e-569c-4908-b5f4-0b6643ae603b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:22.968603 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968534    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a90b92e-569c-4908-b5f4-0b6643ae603b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:22.968603 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968564    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-accelerators-collector-config\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:22.968740 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968607    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a90b92e-569c-4908-b5f4-0b6643ae603b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:22.968740 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968673    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebb63468-4a79-401b-ada4-5b2e1ac38197-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wmrjs\" (UID: \"ebb63468-4a79-401b-ada4-5b2e1ac38197\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:22.968818 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968757    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebb63468-4a79-401b-ada4-5b2e1ac38197-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wmrjs\" (UID: \"ebb63468-4a79-401b-ada4-5b2e1ac38197\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:22.968818 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968801    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a90b92e-569c-4908-b5f4-0b6643ae603b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:22.968890 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968849    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebb63468-4a79-401b-ada4-5b2e1ac38197-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wmrjs\" (UID: \"ebb63468-4a79-401b-ada4-5b2e1ac38197\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:22.968890 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968874    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wllck\" (UniqueName: \"kubernetes.io/projected/ebb63468-4a79-401b-ada4-5b2e1ac38197-kube-api-access-wllck\") pod \"openshift-state-metrics-7bd7489957-wmrjs\" (UID: \"ebb63468-4a79-401b-ada4-5b2e1ac38197\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:22.969014 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968900    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-textfile\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:22.969014 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968923    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-wtmp\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:22.969014 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968957    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61d535d-a6ff-4a10-84af-9c697e073ddb-sys\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:22.969014 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.968994    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chln2\" (UniqueName: \"kubernetes.io/projected/c61d535d-a6ff-4a10-84af-9c697e073ddb-kube-api-access-chln2\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:22.969237 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.969026    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c61d535d-a6ff-4a10-84af-9c697e073ddb-root\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:22.969237 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.969052    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:22.969237 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.969141    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a90b92e-569c-4908-b5f4-0b6643ae603b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:22.969237 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.969166    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-tls\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:22.969237 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.969184    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czpg\" (UniqueName: \"kubernetes.io/projected/3a90b92e-569c-4908-b5f4-0b6643ae603b-kube-api-access-4czpg\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:22.969403 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.969354    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebb63468-4a79-401b-ada4-5b2e1ac38197-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wmrjs\" (UID: \"ebb63468-4a79-401b-ada4-5b2e1ac38197\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:22.972007 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.971983    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebb63468-4a79-401b-ada4-5b2e1ac38197-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wmrjs\" (UID: \"ebb63468-4a79-401b-ada4-5b2e1ac38197\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:22.972837 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.972813    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebb63468-4a79-401b-ada4-5b2e1ac38197-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wmrjs\" (UID: \"ebb63468-4a79-401b-ada4-5b2e1ac38197\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:22.994112 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:22.994069    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wllck\" (UniqueName: \"kubernetes.io/projected/ebb63468-4a79-401b-ada4-5b2e1ac38197-kube-api-access-wllck\") pod \"openshift-state-metrics-7bd7489957-wmrjs\" (UID: \"ebb63468-4a79-401b-ada4-5b2e1ac38197\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:23.035443 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.035381    5072 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:23.070354 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070321    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a90b92e-569c-4908-b5f4-0b6643ae603b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.070502 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070370    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a90b92e-569c-4908-b5f4-0b6643ae603b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.070502 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070395    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-accelerators-collector-config\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.070502 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070438    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a90b92e-569c-4908-b5f4-0b6643ae603b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.070502 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070483    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a90b92e-569c-4908-b5f4-0b6643ae603b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.070737 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070527    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-textfile\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.070737 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070554    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-wtmp\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.070737 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070585    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61d535d-a6ff-4a10-84af-9c697e073ddb-sys\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.070737 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070627    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chln2\" (UniqueName: \"kubernetes.io/projected/c61d535d-a6ff-4a10-84af-9c697e073ddb-kube-api-access-chln2\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.070737 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070657    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c61d535d-a6ff-4a10-84af-9c697e073ddb-root\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.070737 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070684    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.070737 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070733    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a90b92e-569c-4908-b5f4-0b6643ae603b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.071047 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070767    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-tls\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.071047 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070793    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czpg\" (UniqueName: \"kubernetes.io/projected/3a90b92e-569c-4908-b5f4-0b6643ae603b-kube-api-access-4czpg\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.071047 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.070842    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c61d535d-a6ff-4a10-84af-9c697e073ddb-metrics-client-ca\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.071484 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.071462    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c61d535d-a6ff-4a10-84af-9c697e073ddb-metrics-client-ca\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.072019 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.071998    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a90b92e-569c-4908-b5f4-0b6643ae603b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.072327 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.072301    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a90b92e-569c-4908-b5f4-0b6643ae603b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.072739 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.072718    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-accelerators-collector-config\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.073404 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.073350    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-wtmp\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.073503 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.073399    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-textfile\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.073503 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.073409    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61d535d-a6ff-4a10-84af-9c697e073ddb-sys\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.073503 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.073473    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c61d535d-a6ff-4a10-84af-9c697e073ddb-root\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.073895 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.073835    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a90b92e-569c-4908-b5f4-0b6643ae603b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.075907 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.075886    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a90b92e-569c-4908-b5f4-0b6643ae603b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.076371 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.076350    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.076471 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.076405    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a90b92e-569c-4908-b5f4-0b6643ae603b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.077433 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.077413    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c61d535d-a6ff-4a10-84af-9c697e073ddb-node-exporter-tls\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.097825 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.097800    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chln2\" (UniqueName: \"kubernetes.io/projected/c61d535d-a6ff-4a10-84af-9c697e073ddb-kube-api-access-chln2\") pod \"node-exporter-h78rv\" (UID: \"c61d535d-a6ff-4a10-84af-9c697e073ddb\") " pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.098275 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.098255    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czpg\" (UniqueName: \"kubernetes.io/projected/3a90b92e-569c-4908-b5f4-0b6643ae603b-kube-api-access-4czpg\") pod \"kube-state-metrics-5dc8d7d89b-kst59\" (UID: \"3a90b92e-569c-4908-b5f4-0b6643ae603b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.135321 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.135295    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"
Oct 30 03:57:23.196721 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.196688    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"
Oct 30 03:57:23.258422 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.258195    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h78rv"
Oct 30 03:57:23.474254 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.474225    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs"]
Oct 30 03:57:23.475968 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:23.475941    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb63468_4a79_401b_ada4_5b2e1ac38197.slice/crio-295e48951a052caf6b7592bfe4fd169182c66906cc76481748616450eb917df3 WatchSource:0}: Error finding container 295e48951a052caf6b7592bfe4fd169182c66906cc76481748616450eb917df3: Status 404 returned error can't find the container with id 295e48951a052caf6b7592bfe4fd169182c66906cc76481748616450eb917df3
Oct 30 03:57:23.532950 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.532923    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59"]
Oct 30 03:57:23.534186 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:23.534160    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a90b92e_569c_4908_b5f4_0b6643ae603b.slice/crio-0000e908d7eb176cd39402090e175957117ef9bfab3644067e83d66554b79687 WatchSource:0}: Error finding container 0000e908d7eb176cd39402090e175957117ef9bfab3644067e83d66554b79687: Status 404 returned error can't find the container with id 0000e908d7eb176cd39402090e175957117ef9bfab3644067e83d66554b79687
Oct 30 03:57:23.652821 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.652791    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59" event={"ID":"3a90b92e-569c-4908-b5f4-0b6643ae603b","Type":"ContainerStarted","Data":"0000e908d7eb176cd39402090e175957117ef9bfab3644067e83d66554b79687"}
Oct 30 03:57:23.655701 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.655639    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h78rv" event={"ID":"c61d535d-a6ff-4a10-84af-9c697e073ddb","Type":"ContainerStarted","Data":"51ed248605ba082a2257f82be7b58655eaf421e10af6d70fffd2a4900cfb5cf7"}
Oct 30 03:57:23.658043 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.658019    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs" event={"ID":"ebb63468-4a79-401b-ada4-5b2e1ac38197","Type":"ContainerStarted","Data":"490baa40c3a66b6e428d7894fbdfd0d9c56e859f06661194affd5ec52472a6c2"}
Oct 30 03:57:23.658174 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:23.658108    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs" event={"ID":"ebb63468-4a79-401b-ada4-5b2e1ac38197","Type":"ContainerStarted","Data":"295e48951a052caf6b7592bfe4fd169182c66906cc76481748616450eb917df3"}
Oct 30 03:57:24.018100 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.017373    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:57:24.168262 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.168228    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 03:57:24.189897 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.188424    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.203669 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.201290    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6hqf"
Oct 30 03:57:24.213656 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.213558    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 03:57:24.220235 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.219795    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 03:57:24.220235 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.219811    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 03:57:24.220235 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.219924    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 03:57:24.220235 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.219943    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 03:57:24.220235 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.220014    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 03:57:24.220235 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.220108    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 03:57:24.220235 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.220176    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 03:57:24.220235 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.220185    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 03:57:24.220235 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.220107    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pttks\""
Oct 30 03:57:24.220792 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.220251    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 03:57:24.287182 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287105    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.287182 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287155    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.287469 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287184    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.287469 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287276    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.287469 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287310    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.287469 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287347    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4pgj\" (UniqueName: \"kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-kube-api-access-p4pgj\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.287469 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287436    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-volume\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.287469 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287459    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-web-config\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.287803 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287488    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-out\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.287803 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287523    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.287803 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287584    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.287803 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287608    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.287803 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.287630    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.389548 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389507    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-volume\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.389711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389557    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-web-config\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.389711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389586    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-out\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.389711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389620    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.389711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389672    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.389915 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389697    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.389915 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389754    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.389915 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389832    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.389915 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389859    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.389915 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389889    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.390146 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389923    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.390146 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389948    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.390146 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.389988    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4pgj\" (UniqueName: \"kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-kube-api-access-p4pgj\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.392535 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.390377    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.392535 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.390991    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.392535 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.392110    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.393917 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.393847    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.395458 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.395410    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-out\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.396029 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.396008    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-volume\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.396253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.396029    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.397605 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.397583    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-web-config\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.398635 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.398612    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.399303 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.399285    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.403641 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.399948    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.403641 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.403602    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.414894 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.414869    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4pgj\" (UniqueName: \"kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-kube-api-access-p4pgj\") pod \"alertmanager-main-0\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.602044 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.601964    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:57:24.669027 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.668995    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs" event={"ID":"ebb63468-4a79-401b-ada4-5b2e1ac38197","Type":"ContainerStarted","Data":"291ecd221455b1cd3517c997529d71a853ab783a2db9775d738f4fa06ff748f6"}
Oct 30 03:57:24.940963 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.940934    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"]
Oct 30 03:57:24.963587 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.963515    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 03:57:24.963718 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.963620    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:24.970196 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.970170    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"]
Oct 30 03:57:24.987311 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.987285    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 03:57:24.991552 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.987463    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 03:57:24.991552 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.987474    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 03:57:24.991552 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.987687    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 03:57:24.991552 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.987943    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vcktn\""
Oct 30 03:57:24.991552 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.988550    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 03:57:24.991552 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.991544    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7v3v2278e0ivv\""
Oct 30 03:57:24.996115 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.996059    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:24.996237 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.996175    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6snfl\" (UniqueName: \"kubernetes.io/projected/c86a91cd-fdf5-4f67-a404-6011230b0123-kube-api-access-6snfl\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:24.996237 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.996213    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:24.996315 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.996242    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:24.996315 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.996277    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c86a91cd-fdf5-4f67-a404-6011230b0123-metrics-client-ca\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:24.996315 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.996307    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:24.996420 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.996333    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-grpc-tls\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:24.996420 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.996377    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-tls\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:24.996420 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:24.996394    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c86a91cd-fdf5-4f67-a404-6011230b0123-tmp\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.097373 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.097331    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-tls\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.097825 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.097384    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c86a91cd-fdf5-4f67-a404-6011230b0123-tmp\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.097825 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.097448    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.097825 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.097482    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snfl\" (UniqueName: \"kubernetes.io/projected/c86a91cd-fdf5-4f67-a404-6011230b0123-kube-api-access-6snfl\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.097825 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.097507    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.097825 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.097538    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.097825 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.097587    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c86a91cd-fdf5-4f67-a404-6011230b0123-metrics-client-ca\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.097825 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.097628    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.097825 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.097667    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-grpc-tls\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.098539 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.098514    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c86a91cd-fdf5-4f67-a404-6011230b0123-tmp\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.099558 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.099528    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c86a91cd-fdf5-4f67-a404-6011230b0123-metrics-client-ca\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.101794 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.101747    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-tls\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.102717 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.102670    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.103050 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.103020    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.103178 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.103152    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.104139 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.104111    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.104351 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.104305    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c86a91cd-fdf5-4f67-a404-6011230b0123-secret-grpc-tls\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.127112 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.125815    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snfl\" (UniqueName: \"kubernetes.io/projected/c86a91cd-fdf5-4f67-a404-6011230b0123-kube-api-access-6snfl\") pod \"thanos-querier-6948d6f8d4-4vfrk\" (UID: \"c86a91cd-fdf5-4f67-a404-6011230b0123\") " pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.137646 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:25.137601    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497cda50_9fcd_4e3b_b7ab_93d9b599f9a7.slice/crio-0eee56902436eaa9696e722c73086b3cde00c5cd9d2cc47da909b1f5457d27a0 WatchSource:0}: Error finding container 0eee56902436eaa9696e722c73086b3cde00c5cd9d2cc47da909b1f5457d27a0: Status 404 returned error can't find the container with id 0eee56902436eaa9696e722c73086b3cde00c5cd9d2cc47da909b1f5457d27a0
Oct 30 03:57:25.140303 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.140131    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:25.370096 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.370036    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:25.370251 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.370176    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:25.375875 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.375840    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:25.483324 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.483301    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"]
Oct 30 03:57:25.535878 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:25.535848    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86a91cd_fdf5_4f67_a404_6011230b0123.slice/crio-67ef91802faefd183c5b0b7bfb9dfeebc377a0d960325d15e664b1325e1f956b WatchSource:0}: Error finding container 67ef91802faefd183c5b0b7bfb9dfeebc377a0d960325d15e664b1325e1f956b: Status 404 returned error can't find the container with id 67ef91802faefd183c5b0b7bfb9dfeebc377a0d960325d15e664b1325e1f956b
Oct 30 03:57:25.680232 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.680199    5072 generic.go:356] "Generic (PLEG): container finished" podID="c61d535d-a6ff-4a10-84af-9c697e073ddb" containerID="700df5164b9c063375730fc8f1ae8f1f1133f4a7960ba1fb2146f67df11daa39" exitCode=0
Oct 30 03:57:25.680416 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.680385    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h78rv" event={"ID":"c61d535d-a6ff-4a10-84af-9c697e073ddb","Type":"ContainerDied","Data":"700df5164b9c063375730fc8f1ae8f1f1133f4a7960ba1fb2146f67df11daa39"}
Oct 30 03:57:25.685820 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.685775    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs" event={"ID":"ebb63468-4a79-401b-ada4-5b2e1ac38197","Type":"ContainerStarted","Data":"98cdd9463177e0d6ee68eff358f8ab3278e6255cef74b9bb4cbaa61ac1e90e0e"}
Oct 30 03:57:25.689221 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.689191    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk" event={"ID":"c86a91cd-fdf5-4f67-a404-6011230b0123","Type":"ContainerStarted","Data":"67ef91802faefd183c5b0b7bfb9dfeebc377a0d960325d15e664b1325e1f956b"}
Oct 30 03:57:25.691033 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.691001    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerStarted","Data":"0eee56902436eaa9696e722c73086b3cde00c5cd9d2cc47da909b1f5457d27a0"}
Oct 30 03:57:25.693398 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.693366    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59" event={"ID":"3a90b92e-569c-4908-b5f4-0b6643ae603b","Type":"ContainerStarted","Data":"443caecabe3d8f98aecc736e4c5059b59999aa19e6fa068070ed116afc87d0e3"}
Oct 30 03:57:25.700135 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.700113    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:25.818313 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:25.818250    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wmrjs" podStartSLOduration=1.9339909130000001 podStartE2EDuration="3.818229288s" totalImagesPullingTime="1.884238375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:25.816735757 +0000 UTC m=+25.558158855" watchObservedRunningTime="2026-10-30 03:57:25.818229288 +0000 UTC m=+25.559652387"
Oct 30 03:57:26.704729 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:26.704692    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59" event={"ID":"3a90b92e-569c-4908-b5f4-0b6643ae603b","Type":"ContainerStarted","Data":"375b22bacb720f988559009ca2cb627736b9672d30cf1c16906f7ac9d1337a43"}
Oct 30 03:57:26.705299 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:26.704755    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59" event={"ID":"3a90b92e-569c-4908-b5f4-0b6643ae603b","Type":"ContainerStarted","Data":"6bd896e4a287e473d3562791c7264f3b62a5368290792b870b1e486994f54e42"}
Oct 30 03:57:26.710627 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:26.710594    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h78rv" event={"ID":"c61d535d-a6ff-4a10-84af-9c697e073ddb","Type":"ContainerStarted","Data":"2333f1781b47e9d23fb41b441fb791e472890cec040cb74b02191628ce1e389e"}
Oct 30 03:57:26.710741 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:26.710648    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h78rv" event={"ID":"c61d535d-a6ff-4a10-84af-9c697e073ddb","Type":"ContainerStarted","Data":"73581440f11c9ee02392d90ea7ce50be489a0353335262a5cc665d1b5597e814"}
Oct 30 03:57:26.749985 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:26.749925    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-kst59" podStartSLOduration=2.7194467700000002 podStartE2EDuration="4.749912298s" totalImagesPullingTime="2.030465528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:26.749468349 +0000 UTC m=+26.490891446" watchObservedRunningTime="2026-10-30 03:57:26.749912298 +0000 UTC m=+26.491335386"
Oct 30 03:57:26.798761 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:26.796639    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h78rv" podStartSLOduration=2.763312624 podStartE2EDuration="4.796622153s" totalImagesPullingTime="1.921994753s" totalInitContainerRuntime="111.314776ms" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:26.793954631 +0000 UTC m=+26.535377721" watchObservedRunningTime="2026-10-30 03:57:26.796622153 +0000 UTC m=+26.538045251"
Oct 30 03:57:27.134271 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.134238    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:27.242932 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.242889    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"]
Oct 30 03:57:27.263212 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.263177    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.267727 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.267662    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"]
Oct 30 03:57:27.279569 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.279542    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 03:57:27.279723 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.279699    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 03:57:27.279787 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.279741    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 03:57:27.279835 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.279795    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vpm8b\""
Oct 30 03:57:27.279889 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.279872    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-frbmm5f0dsvi0\""
Oct 30 03:57:27.279939 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.279894    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 03:57:27.330152 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.330106    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed7968f8-8483-44d3-bd36-201b7f0bde51-secret-metrics-server-client-certs\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.330306 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.330214    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed7968f8-8483-44d3-bd36-201b7f0bde51-secret-metrics-server-tls\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.330368 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.330318    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed7968f8-8483-44d3-bd36-201b7f0bde51-audit-log\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.330368 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.330353    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88pr\" (UniqueName: \"kubernetes.io/projected/ed7968f8-8483-44d3-bd36-201b7f0bde51-kube-api-access-k88pr\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.330496 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.330382    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed7968f8-8483-44d3-bd36-201b7f0bde51-metrics-server-audit-profiles\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.330496 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.330448    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7968f8-8483-44d3-bd36-201b7f0bde51-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.330496 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.330478    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed7968f8-8483-44d3-bd36-201b7f0bde51-client-ca-bundle\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.432859 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.431717    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed7968f8-8483-44d3-bd36-201b7f0bde51-audit-log\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.432859 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.431770    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88pr\" (UniqueName: \"kubernetes.io/projected/ed7968f8-8483-44d3-bd36-201b7f0bde51-kube-api-access-k88pr\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.432859 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.431798    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed7968f8-8483-44d3-bd36-201b7f0bde51-metrics-server-audit-profiles\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.432859 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.431854    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7968f8-8483-44d3-bd36-201b7f0bde51-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.432859 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.431879    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed7968f8-8483-44d3-bd36-201b7f0bde51-client-ca-bundle\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.432859 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.431945    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed7968f8-8483-44d3-bd36-201b7f0bde51-secret-metrics-server-client-certs\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.432859 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.431980    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed7968f8-8483-44d3-bd36-201b7f0bde51-secret-metrics-server-tls\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.433975 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.433232    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed7968f8-8483-44d3-bd36-201b7f0bde51-audit-log\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.433975 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.433355    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7968f8-8483-44d3-bd36-201b7f0bde51-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.433975 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.433721    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed7968f8-8483-44d3-bd36-201b7f0bde51-metrics-server-audit-profiles\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.435921 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.435895    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed7968f8-8483-44d3-bd36-201b7f0bde51-client-ca-bundle\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.436019 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.435973    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed7968f8-8483-44d3-bd36-201b7f0bde51-secret-metrics-server-tls\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.437041 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.437016    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed7968f8-8483-44d3-bd36-201b7f0bde51-secret-metrics-server-client-certs\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.458717 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.458668    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88pr\" (UniqueName: \"kubernetes.io/projected/ed7968f8-8483-44d3-bd36-201b7f0bde51-kube-api-access-k88pr\") pod \"metrics-server-7858fcf8fc-qc6gr\" (UID: \"ed7968f8-8483-44d3-bd36-201b7f0bde51\") " pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.577485 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.577397    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"]
Oct 30 03:57:27.602702 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.602423    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:27.612476 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.612420    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"]
Oct 30 03:57:27.612622 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.612538    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"
Oct 30 03:57:27.629363 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.629340    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pg9q9\""
Oct 30 03:57:27.629560 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.629543    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 03:57:27.634250 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.634224    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkwxh\" (UniqueName: \"kubernetes.io/projected/750f1340-7214-425f-8511-84fa94adb834-kube-api-access-jkwxh\") pod \"monitoring-plugin-77f4f76594-hkgg9\" (UID: \"750f1340-7214-425f-8511-84fa94adb834\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"
Oct 30 03:57:27.634364 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.634269    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/750f1340-7214-425f-8511-84fa94adb834-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hkgg9\" (UID: \"750f1340-7214-425f-8511-84fa94adb834\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"
Oct 30 03:57:27.719870 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.719843    5072 generic.go:356] "Generic (PLEG): container finished" podID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerID="38420e8d51a6b021c1f94f1ac6b973d3b0a5a0ae377602280ea1b877c2758199" exitCode=0
Oct 30 03:57:27.720321 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.719971    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerDied","Data":"38420e8d51a6b021c1f94f1ac6b973d3b0a5a0ae377602280ea1b877c2758199"}
Oct 30 03:57:27.735336 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.734736    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkwxh\" (UniqueName: \"kubernetes.io/projected/750f1340-7214-425f-8511-84fa94adb834-kube-api-access-jkwxh\") pod \"monitoring-plugin-77f4f76594-hkgg9\" (UID: \"750f1340-7214-425f-8511-84fa94adb834\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"
Oct 30 03:57:27.735336 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.734766    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/750f1340-7214-425f-8511-84fa94adb834-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hkgg9\" (UID: \"750f1340-7214-425f-8511-84fa94adb834\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"
Oct 30 03:57:27.738379 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.738350    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/750f1340-7214-425f-8511-84fa94adb834-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hkgg9\" (UID: \"750f1340-7214-425f-8511-84fa94adb834\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"
Oct 30 03:57:27.759637 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.759343    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkwxh\" (UniqueName: \"kubernetes.io/projected/750f1340-7214-425f-8511-84fa94adb834-kube-api-access-jkwxh\") pod \"monitoring-plugin-77f4f76594-hkgg9\" (UID: \"750f1340-7214-425f-8511-84fa94adb834\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"
Oct 30 03:57:27.950987 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.950955    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"]
Oct 30 03:57:27.952322 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:27.952292    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7968f8_8483_44d3_bd36_201b7f0bde51.slice/crio-38f0fb550fb8dcabb9ba4e33adf0caebb08a6e7477237f5319b8175cd98b9797 WatchSource:0}: Error finding container 38f0fb550fb8dcabb9ba4e33adf0caebb08a6e7477237f5319b8175cd98b9797: Status 404 returned error can't find the container with id 38f0fb550fb8dcabb9ba4e33adf0caebb08a6e7477237f5319b8175cd98b9797
Oct 30 03:57:27.974139 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:27.974117    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"
Oct 30 03:57:28.067572 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.066282    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"]
Oct 30 03:57:28.096460 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.091760    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.106443 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.105025    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"]
Oct 30 03:57:28.115711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.115479    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 03:57:28.115711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.115507    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 03:57:28.115711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.115523    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-b55jw\""
Oct 30 03:57:28.115711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.115574    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 03:57:28.115711 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.115614    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 03:57:28.129012 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.128985    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 03:57:28.129181 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.129035    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 03:57:28.139492 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.139451    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b142444b-c75c-480d-83bf-54350d59439c-secret-telemeter-client\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.139625 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.139559    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b142444b-c75c-480d-83bf-54350d59439c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.139625 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.139595    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b142444b-c75c-480d-83bf-54350d59439c-metrics-client-ca\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.139739 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.139705    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b142444b-c75c-480d-83bf-54350d59439c-telemeter-client-tls\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.139827 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.139807    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kghd6\" (UniqueName: \"kubernetes.io/projected/b142444b-c75c-480d-83bf-54350d59439c-kube-api-access-kghd6\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.139891 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.139849    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b142444b-c75c-480d-83bf-54350d59439c-federate-client-tls\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.139891 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.139882    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b142444b-c75c-480d-83bf-54350d59439c-serving-certs-ca-bundle\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.139993 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.139973    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b142444b-c75c-480d-83bf-54350d59439c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.241320 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.241231    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b142444b-c75c-480d-83bf-54350d59439c-telemeter-client-tls\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.241320 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.241309    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghd6\" (UniqueName: \"kubernetes.io/projected/b142444b-c75c-480d-83bf-54350d59439c-kube-api-access-kghd6\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.241537 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.241339    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b142444b-c75c-480d-83bf-54350d59439c-federate-client-tls\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.241537 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.241378    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b142444b-c75c-480d-83bf-54350d59439c-serving-certs-ca-bundle\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.241537 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.241431    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b142444b-c75c-480d-83bf-54350d59439c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.242198 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.242167    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b142444b-c75c-480d-83bf-54350d59439c-secret-telemeter-client\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.242349 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.242222    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b142444b-c75c-480d-83bf-54350d59439c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.242349 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.242249    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b142444b-c75c-480d-83bf-54350d59439c-metrics-client-ca\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.243243 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.243212    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b142444b-c75c-480d-83bf-54350d59439c-metrics-client-ca\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.243820 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.243772    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b142444b-c75c-480d-83bf-54350d59439c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.244896 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.244869    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b142444b-c75c-480d-83bf-54350d59439c-federate-client-tls\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.245196 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.245177    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b142444b-c75c-480d-83bf-54350d59439c-serving-certs-ca-bundle\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.245937 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.245888    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b142444b-c75c-480d-83bf-54350d59439c-telemeter-client-tls\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.246571 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.246534    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b142444b-c75c-480d-83bf-54350d59439c-secret-telemeter-client\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.246674 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.246645    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b142444b-c75c-480d-83bf-54350d59439c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.266546 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.266522    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghd6\" (UniqueName: \"kubernetes.io/projected/b142444b-c75c-480d-83bf-54350d59439c-kube-api-access-kghd6\") pod \"telemeter-client-78d6d46cf9-8zhdq\" (UID: \"b142444b-c75c-480d-83bf-54350d59439c\") " pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.302724 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.302657    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"]
Oct 30 03:57:28.335445 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:28.335408    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750f1340_7214_425f_8511_84fa94adb834.slice/crio-0c5418aea316911173a6e98001ccad216ad0f69915d1c0405813e2baffe40e3f WatchSource:0}: Error finding container 0c5418aea316911173a6e98001ccad216ad0f69915d1c0405813e2baffe40e3f: Status 404 returned error can't find the container with id 0c5418aea316911173a6e98001ccad216ad0f69915d1c0405813e2baffe40e3f
Oct 30 03:57:28.454314 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.454126    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"
Oct 30 03:57:28.730811 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.730674    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9" event={"ID":"750f1340-7214-425f-8511-84fa94adb834","Type":"ContainerStarted","Data":"0c5418aea316911173a6e98001ccad216ad0f69915d1c0405813e2baffe40e3f"}
Oct 30 03:57:28.737916 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.737316    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk" event={"ID":"c86a91cd-fdf5-4f67-a404-6011230b0123","Type":"ContainerStarted","Data":"7859bf3cdd670b20a9f660eaa2db36f780217dac1abff9abe5cc69f45e779abe"}
Oct 30 03:57:28.737916 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.737404    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk" event={"ID":"c86a91cd-fdf5-4f67-a404-6011230b0123","Type":"ContainerStarted","Data":"3ce671356f5594dd662d3d46df0bd6f473da1a2ba8cefc05f270661f66481186"}
Oct 30 03:57:28.737916 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.737448    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk" event={"ID":"c86a91cd-fdf5-4f67-a404-6011230b0123","Type":"ContainerStarted","Data":"6b88ae55067bcd5f1bb4504138c0ff44372156a41a07f97b01a9c52f607afa94"}
Oct 30 03:57:28.740695 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.740629    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr" event={"ID":"ed7968f8-8483-44d3-bd36-201b7f0bde51","Type":"ContainerStarted","Data":"38f0fb550fb8dcabb9ba4e33adf0caebb08a6e7477237f5319b8175cd98b9797"}
Oct 30 03:57:28.838527 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:28.837246    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq"]
Oct 30 03:57:28.841094 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:28.841032    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb142444b_c75c_480d_83bf_54350d59439c.slice/crio-405bda99c3b8c40bf8a95f083349f2e678634c5539a800fad691ebb2c6f9fd7f WatchSource:0}: Error finding container 405bda99c3b8c40bf8a95f083349f2e678634c5539a800fad691ebb2c6f9fd7f: Status 404 returned error can't find the container with id 405bda99c3b8c40bf8a95f083349f2e678634c5539a800fad691ebb2c6f9fd7f
Oct 30 03:57:29.264624 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.264583    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:29.265644 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.265622    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:29.275753 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.275507    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:29.762867 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.762830    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq" event={"ID":"b142444b-c75c-480d-83bf-54350d59439c","Type":"ContainerStarted","Data":"405bda99c3b8c40bf8a95f083349f2e678634c5539a800fad691ebb2c6f9fd7f"}
Oct 30 03:57:29.769027 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.769002    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:57:29.841451 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.837026    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 03:57:29.879116 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.878924    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c4fcd595-5xn9h"]
Oct 30 03:57:29.879387 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.879367    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.881821 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.880868    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 03:57:29.898173 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.898143    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 03:57:29.898303 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.898223    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 03:57:29.898376 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.898309    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 03:57:29.898432 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.898388    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 03:57:29.898488 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.898456    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 03:57:29.898536 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.898514    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 03:57:29.898591 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.898578    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 03:57:29.902065 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.902045    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 03:57:29.911435 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.911409    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 03:57:29.911545 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.911481    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 03:57:29.911545 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.911538    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 03:57:29.911827 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.911804    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9c3566kc47c63\""
Oct 30 03:57:29.911906 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.911888    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 03:57:29.911965 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.911951    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7cnb2\""
Oct 30 03:57:29.940538 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.938638    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 03:57:29.940538 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.938913    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.975897    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.975951    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.975982    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976011    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976091    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976117    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-config\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976143    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7sp\" (UniqueName: \"kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-kube-api-access-5h7sp\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976181    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-config-out\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976205    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976263    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976291    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976314    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976354    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976385    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-web-config\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976427    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976483    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976514    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976535    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976562    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:29.976919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:29.976587    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.079402 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.078126    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.079402 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.078172    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.079402 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.078197    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.079402 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.078227    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.079402 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.078254    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.079402 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.078289    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.079402 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.078947    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.079402 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.079169    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.079907 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.079633    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.078330    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080034    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080070    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080159    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080189    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-config\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080218    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7sp\" (UniqueName: \"kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-kube-api-access-5h7sp\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080262    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-config-out\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080289    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080343    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080362    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080383    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080430    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080449    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-web-config\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.080817 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.080473    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.083753 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.082884    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.084708 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.084683    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.084952 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.084821    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.085327 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.085219    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.085490 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.085382    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-config-out\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.085759 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.085736    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.086584 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.086538    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.089921 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.087769    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.090010 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.088665    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.090010 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.088839    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.090147 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.090116    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-config\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.095301 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.095279    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.113356 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.113323    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7sp\" (UniqueName: \"kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-kube-api-access-5h7sp\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.122688 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.122662    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.122809 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.122737    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.122879 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.122831    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.124636 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.124593    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-web-config\") pod \"prometheus-k8s-0\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:30.342026 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:30.340464    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:31.110887 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.110855    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 03:57:31.297032 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:31.296995    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd8b824_8c21_40aa_9a1d_f3913690e556.slice/crio-113c98977b1cf937aab0fd80af518edf2c2091682e6adf2e92e6a776060635c7 WatchSource:0}: Error finding container 113c98977b1cf937aab0fd80af518edf2c2091682e6adf2e92e6a776060635c7: Status 404 returned error can't find the container with id 113c98977b1cf937aab0fd80af518edf2c2091682e6adf2e92e6a776060635c7
Oct 30 03:57:31.779197 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.779135    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr" event={"ID":"ed7968f8-8483-44d3-bd36-201b7f0bde51","Type":"ContainerStarted","Data":"dce0c2b29045de8468704a845c28a300205389b555fa788626b624c91fcc119c"}
Oct 30 03:57:31.781772 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.781735    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9" event={"ID":"750f1340-7214-425f-8511-84fa94adb834","Type":"ContainerStarted","Data":"0d8e3a2dbe4fd32943a9ac0fc1f175b5a2edc8e497c9ae20266c3b78aa822568"}
Oct 30 03:57:31.781979 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.781961    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"
Oct 30 03:57:31.787469 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.787440    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk" event={"ID":"c86a91cd-fdf5-4f67-a404-6011230b0123","Type":"ContainerStarted","Data":"c455423fd29c1478f9ae6f21765a9c0c3398997e343b8a424451f8dc96d6aebd"}
Oct 30 03:57:31.787712 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.787512    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk" event={"ID":"c86a91cd-fdf5-4f67-a404-6011230b0123","Type":"ContainerStarted","Data":"85bc4509e62591e8052dc77c10f98d51ff49ce97a81c6bd49f5263be90ed05ce"}
Oct 30 03:57:31.787712 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.787549    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk" event={"ID":"c86a91cd-fdf5-4f67-a404-6011230b0123","Type":"ContainerStarted","Data":"8ad151a384f0b877d1108a59120a08082746e9b3a94ebfc4e4afda1b95b59293"}
Oct 30 03:57:31.787712 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.787695    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:31.790814 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.790791    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9"
Oct 30 03:57:31.793044 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.792823    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerStarted","Data":"51a9cd58223c187e09f26eaf662519381e049886ed152e06645a799da125f844"}
Oct 30 03:57:31.793044 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.792905    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerStarted","Data":"85d56d9fdcf727fbf293ab4d01151630c986889a5b1b3b6fd381db6dd532ed5d"}
Oct 30 03:57:31.793044 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.792947    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerStarted","Data":"65118ece72d44bb45ae90e8d701aa7168cccdfb385ce75bc691e0ff655146a84"}
Oct 30 03:57:31.793044 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.792981    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerStarted","Data":"9ffedb59c6cff95a4c7d6c93797e5b9d8f90e63e8b784407d7f80c48bee4d296"}
Oct 30 03:57:31.795821 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.795798    5072 generic.go:356] "Generic (PLEG): container finished" podID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerID="29276ff4a38cbe9f9cc9c9ab8a05e7b4a3993af4e124a07d73bb9d8bd1b19c42" exitCode=0
Oct 30 03:57:31.797067 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.797044    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerDied","Data":"29276ff4a38cbe9f9cc9c9ab8a05e7b4a3993af4e124a07d73bb9d8bd1b19c42"}
Oct 30 03:57:31.797200 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.797130    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerStarted","Data":"113c98977b1cf937aab0fd80af518edf2c2091682e6adf2e92e6a776060635c7"}
Oct 30 03:57:31.827013 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.826364    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr" podStartSLOduration=1.438832578 podStartE2EDuration="4.826346706s" totalImagesPullingTime="3.387514128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:31.824600463 +0000 UTC m=+31.566023561" watchObservedRunningTime="2026-10-30 03:57:31.826346706 +0000 UTC m=+31.567769805"
Oct 30 03:57:31.869756 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.869655    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hkgg9" podStartSLOduration=1.890207604 podStartE2EDuration="4.86963736s" totalImagesPullingTime="2.979429756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:31.868330207 +0000 UTC m=+31.609753303" watchObservedRunningTime="2026-10-30 03:57:31.86963736 +0000 UTC m=+31.611060460"
Oct 30 03:57:31.935924 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.935889    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-75c989c798-7tc4c"]
Oct 30 03:57:31.953341 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.953314    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:31.966786 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.966758    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c989c798-7tc4c"]
Oct 30 03:57:31.981748 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:31.981690    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk" podStartSLOduration=3.129761036 podStartE2EDuration="7.981671233s" totalImagesPullingTime="4.851910197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:31.980391506 +0000 UTC m=+31.721814604" watchObservedRunningTime="2026-10-30 03:57:31.981671233 +0000 UTC m=+31.723094330"
Oct 30 03:57:32.001946 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.001911    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-serving-cert\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.002116 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.002002    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-oauth-serving-cert\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.002116 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.002035    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4n6l\" (UniqueName: \"kubernetes.io/projected/fe0bb808-beaa-4907-94e0-29b68fe8d121-kube-api-access-x4n6l\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.002238 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.002112    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe0bb808-beaa-4907-94e0-29b68fe8d121-tmp\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.002327 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.002290    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-oauth-config\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.002498 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.002475    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-config\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.002607 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.002522    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-service-ca\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.002607 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.002579    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-trusted-ca-bundle\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.103328 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.103284    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4n6l\" (UniqueName: \"kubernetes.io/projected/fe0bb808-beaa-4907-94e0-29b68fe8d121-kube-api-access-x4n6l\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.103421 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.103354    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe0bb808-beaa-4907-94e0-29b68fe8d121-tmp\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.104268 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.103534    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-oauth-config\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.104268 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.103573    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-config\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.104268 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.103617    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-service-ca\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.104268 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.103650    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-trusted-ca-bundle\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.104268 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.103736    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-serving-cert\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.104268 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.103861    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe0bb808-beaa-4907-94e0-29b68fe8d121-tmp\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.104268 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.103984    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-oauth-serving-cert\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.104855 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.104760    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-oauth-serving-cert\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.104855 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.104811    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-config\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.104987 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.104974    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-trusted-ca-bundle\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.105392 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.105349    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-service-ca\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.107558 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.107539    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-oauth-config\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.110040 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.110020    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-serving-cert\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.130287 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.130262    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4n6l\" (UniqueName: \"kubernetes.io/projected/fe0bb808-beaa-4907-94e0-29b68fe8d121-kube-api-access-x4n6l\") pod \"console-75c989c798-7tc4c\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") " pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.295170 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.295138    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:32.636409 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.636378    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c989c798-7tc4c"]
Oct 30 03:57:32.638499 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:32.638470    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0bb808_beaa_4907_94e0_29b68fe8d121.slice/crio-2bd04fade11615a37e724d4e01c7fd9093d3a3e1af655dc800cd0b80d921f4c0 WatchSource:0}: Error finding container 2bd04fade11615a37e724d4e01c7fd9093d3a3e1af655dc800cd0b80d921f4c0: Status 404 returned error can't find the container with id 2bd04fade11615a37e724d4e01c7fd9093d3a3e1af655dc800cd0b80d921f4c0
Oct 30 03:57:32.808799 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.807837    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq" event={"ID":"b142444b-c75c-480d-83bf-54350d59439c","Type":"ContainerStarted","Data":"e9869e6719ad20b867c097d5669e4f15efb7e49014033460640c99777e2b65d9"}
Oct 30 03:57:32.808799 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.807904    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq" event={"ID":"b142444b-c75c-480d-83bf-54350d59439c","Type":"ContainerStarted","Data":"a2353638151d825a91f59b8fe0dacae290b52fae4ed52be2d5238997444afc43"}
Oct 30 03:57:32.808799 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.807924    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq" event={"ID":"b142444b-c75c-480d-83bf-54350d59439c","Type":"ContainerStarted","Data":"975d9bc6b3a8739f13ad606ebc7d77e5fcbaeaca3d7cb4ec65e9b95a868318ef"}
Oct 30 03:57:32.813473 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.813380    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c989c798-7tc4c" event={"ID":"fe0bb808-beaa-4907-94e0-29b68fe8d121","Type":"ContainerStarted","Data":"86c475a381ec3f294c4dbf871c898d9e6b8e90cd2ed7f28d345a1d61a7446c28"}
Oct 30 03:57:32.813473 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.813441    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c989c798-7tc4c" event={"ID":"fe0bb808-beaa-4907-94e0-29b68fe8d121","Type":"ContainerStarted","Data":"2bd04fade11615a37e724d4e01c7fd9093d3a3e1af655dc800cd0b80d921f4c0"}
Oct 30 03:57:32.821150 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.821027    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerStarted","Data":"7d682b87dc3fb7441e82b4bf0746df95b44564ec82aa09f48ef2dae143ef6a42"}
Oct 30 03:57:32.821150 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.821098    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerStarted","Data":"91390c129310e4a3962d846caf83e4ea0984faf224cc92db1d9e3c2ea7287bdb"}
Oct 30 03:57:32.858903 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.858464    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-78d6d46cf9-8zhdq" podStartSLOduration=1.739785998 podStartE2EDuration="4.85844423s" totalImagesPullingTime="3.118658232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:32.857847156 +0000 UTC m=+32.599270258" watchObservedRunningTime="2026-10-30 03:57:32.85844423 +0000 UTC m=+32.599867329"
Oct 30 03:57:32.966445 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.965668    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.055017898 podStartE2EDuration="8.965653531s" totalImagesPullingTime="4.905923594s" totalInitContainerRuntime="4.712039ms" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:32.963561126 +0000 UTC m=+32.704984258" watchObservedRunningTime="2026-10-30 03:57:32.965653531 +0000 UTC m=+32.707076628"
Oct 30 03:57:32.966445 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:32.965818    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-75c989c798-7tc4c" podStartSLOduration=1.965813232 podStartE2EDuration="1.965813232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:32.905131319 +0000 UTC m=+32.646554418" watchObservedRunningTime="2026-10-30 03:57:32.965813232 +0000 UTC m=+32.707236329"
Oct 30 03:57:34.742043 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.741964    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c989c798-7tc4c"]
Oct 30 03:57:34.774682 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.774652    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-869d9c4d7-49r74"]
Oct 30 03:57:34.810983 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.810944    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-869d9c4d7-49r74"]
Oct 30 03:57:34.811171 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.811070    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.840733 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.840695    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84436e2-c830-4134-af93-5fc2f934ccbf-tmp\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.840864 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.840825    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-trusted-ca-bundle\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.840935 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.840869    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-oauth-serving-cert\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.840935 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.840905    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-console-config\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.841037 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.840940    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-service-ca\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.841037 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.840971    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-oauth-config\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.841037 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.841032    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-serving-cert\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.841224 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.841063    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sntl5\" (UniqueName: \"kubernetes.io/projected/f84436e2-c830-4134-af93-5fc2f934ccbf-kube-api-access-sntl5\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.931995 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.931966    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k2wgk"
Oct 30 03:57:34.944053 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.942938    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-trusted-ca-bundle\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.944053 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.942976    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-oauth-serving-cert\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.944053 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.942995    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-console-config\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.944053 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.943115    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-service-ca\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.944053 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.943156    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-oauth-config\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.944053 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.943218    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-serving-cert\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.944053 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.943251    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sntl5\" (UniqueName: \"kubernetes.io/projected/f84436e2-c830-4134-af93-5fc2f934ccbf-kube-api-access-sntl5\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.944053 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.943311    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84436e2-c830-4134-af93-5fc2f934ccbf-tmp\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.944053 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.943735    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-console-config\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.944053 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.943799    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-oauth-serving-cert\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.944053 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.943836    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-trusted-ca-bundle\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.944827 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.944399    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-service-ca\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.944827 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.944693    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84436e2-c830-4134-af93-5fc2f934ccbf-tmp\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.947124 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.947098    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-oauth-config\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.948714 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.947810    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-serving-cert\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:34.968761 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:34.968736    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sntl5\" (UniqueName: \"kubernetes.io/projected/f84436e2-c830-4134-af93-5fc2f934ccbf-kube-api-access-sntl5\") pod \"console-869d9c4d7-49r74\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") " pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:35.144119 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:35.144014    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:35.517219 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:35.517031    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-869d9c4d7-49r74"]
Oct 30 03:57:35.519097 ip-10-0-139-99 kubenswrapper[5072]: W1030 03:57:35.519041    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84436e2_c830_4134_af93_5fc2f934ccbf.slice/crio-020d00fb0ee700473b02c072e0bf1339e5de17e52c2972317e21ce7812e05d8e WatchSource:0}: Error finding container 020d00fb0ee700473b02c072e0bf1339e5de17e52c2972317e21ce7812e05d8e: Status 404 returned error can't find the container with id 020d00fb0ee700473b02c072e0bf1339e5de17e52c2972317e21ce7812e05d8e
Oct 30 03:57:35.843890 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:35.842493    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869d9c4d7-49r74" event={"ID":"f84436e2-c830-4134-af93-5fc2f934ccbf","Type":"ContainerStarted","Data":"bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26"}
Oct 30 03:57:35.843890 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:35.842560    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869d9c4d7-49r74" event={"ID":"f84436e2-c830-4134-af93-5fc2f934ccbf","Type":"ContainerStarted","Data":"020d00fb0ee700473b02c072e0bf1339e5de17e52c2972317e21ce7812e05d8e"}
Oct 30 03:57:36.871013 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:36.870948    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerStarted","Data":"d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648"}
Oct 30 03:57:36.871378 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:36.871026    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerStarted","Data":"96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5"}
Oct 30 03:57:37.611405 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:37.611329    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6878899c59-ksp2c"
Oct 30 03:57:37.686920 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:37.686859    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-869d9c4d7-49r74" podStartSLOduration=3.686839131 podStartE2EDuration="3.686839131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:35.891262283 +0000 UTC m=+35.632685379" watchObservedRunningTime="2026-10-30 03:57:37.686839131 +0000 UTC m=+37.428262230"
Oct 30 03:57:37.708481 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:37.708440    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bbcfd588b-rlkpf"]
Oct 30 03:57:37.826957 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:37.825984    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6948d6f8d4-4vfrk"
Oct 30 03:57:37.924048 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:37.923968    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerStarted","Data":"2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4"}
Oct 30 03:57:37.924521 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:37.924053    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerStarted","Data":"9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb"}
Oct 30 03:57:37.924521 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:37.924128    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerStarted","Data":"350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464"}
Oct 30 03:57:37.924521 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:37.924169    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerStarted","Data":"1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a"}
Oct 30 03:57:37.992056 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:37.991989    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.413575478 podStartE2EDuration="8.991968393s" totalImagesPullingTime="4.570160192s" totalInitContainerRuntime="8.232723ms" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:37.991632788 +0000 UTC m=+37.733055887" watchObservedRunningTime="2026-10-30 03:57:37.991968393 +0000 UTC m=+37.733391492"
Oct 30 03:57:39.944982 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:39.944894    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j55cl" event={"ID":"5bb6afde-1ba7-4045-841d-528387cf7f9d","Type":"ContainerStarted","Data":"98cee4d9a4156cca94cd37a0ece5769bd3611ab0439b1b6ba9b0e3401e7f15da"}
Oct 30 03:57:39.945526 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:39.945264    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-j55cl"
Oct 30 03:57:39.961649 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:39.961620    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-j55cl"
Oct 30 03:57:40.064217 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.064160    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-j55cl" podStartSLOduration=1.9759199139999999 podStartE2EDuration="25.064139793s" totalImagesPullingTime="23.088219879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:40.019409377 +0000 UTC m=+39.760832475" watchObservedRunningTime="2026-10-30 03:57:40.064139793 +0000 UTC m=+39.805562889"
Oct 30 03:57:40.068272 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.068228    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f" podUID="395d5cbd-c077-4578-b377-3034e42a2b94" containerName="registry" containerID="cri-o://833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd" gracePeriod=30
Oct 30 03:57:40.343416 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.343377    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:40.597618 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.597586    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:40.620925 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.620890    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-registry-certificates\") pod \"395d5cbd-c077-4578-b377-3034e42a2b94\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") "
Oct 30 03:57:40.622642 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.620966    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-kube-api-access-99bzl\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-kube-api-access-99bzl\") pod \"395d5cbd-c077-4578-b377-3034e42a2b94\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") "
Oct 30 03:57:40.622642 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.621032    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-installation-pull-secrets\") pod \"395d5cbd-c077-4578-b377-3034e42a2b94\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") "
Oct 30 03:57:40.622642 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.621061    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-trusted-ca\") pod \"395d5cbd-c077-4578-b377-3034e42a2b94\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") "
Oct 30 03:57:40.622642 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.621142    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-bound-sa-token\") pod \"395d5cbd-c077-4578-b377-3034e42a2b94\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") "
Oct 30 03:57:40.622642 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.621164    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/395d5cbd-c077-4578-b377-3034e42a2b94-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/395d5cbd-c077-4578-b377-3034e42a2b94-ca-trust-extracted\") pod \"395d5cbd-c077-4578-b377-3034e42a2b94\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") "
Oct 30 03:57:40.622642 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.621215    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-image-registry-private-configuration\") pod \"395d5cbd-c077-4578-b377-3034e42a2b94\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") "
Oct 30 03:57:40.622642 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.621289    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") pod \"395d5cbd-c077-4578-b377-3034e42a2b94\" (UID: \"395d5cbd-c077-4578-b377-3034e42a2b94\") "
Oct 30 03:57:40.622642 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.621426    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-registry-certificates" pod "395d5cbd-c077-4578-b377-3034e42a2b94" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:40.622642 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.621740    5072 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-registry-certificates\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:40.633238 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.629478    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-trusted-ca" pod "395d5cbd-c077-4578-b377-3034e42a2b94" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:40.633238 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.632243    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls" pod "395d5cbd-c077-4578-b377-3034e42a2b94" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:40.633238 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.633176    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-installation-pull-secrets" pod "395d5cbd-c077-4578-b377-3034e42a2b94" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:40.633238 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.633213    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-bound-sa-token" pod "395d5cbd-c077-4578-b377-3034e42a2b94" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:40.639177 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.636112    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-image-registry-private-configuration" pod "395d5cbd-c077-4578-b377-3034e42a2b94" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:40.639744 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.639709    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395d5cbd-c077-4578-b377-3034e42a2b94-ca-trust-extracted" pod "395d5cbd-c077-4578-b377-3034e42a2b94" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:40.659312 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.659276    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-kube-api-access-99bzl" pod "395d5cbd-c077-4578-b377-3034e42a2b94" (UID: "395d5cbd-c077-4578-b377-3034e42a2b94"). InnerVolumeSpecName "kube-api-access-99bzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:40.723165 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.723093    5072 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-image-registry-private-configuration\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:40.723165 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.723134    5072 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-registry-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:40.723165 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.723153    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99bzl\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-kube-api-access-99bzl\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:40.723165 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.723170    5072 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/395d5cbd-c077-4578-b377-3034e42a2b94-installation-pull-secrets\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:40.723495 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.723185    5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/395d5cbd-c077-4578-b377-3034e42a2b94-trusted-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:40.723495 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.723197    5072 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/395d5cbd-c077-4578-b377-3034e42a2b94-bound-sa-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:40.723495 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.723210    5072 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/395d5cbd-c077-4578-b377-3034e42a2b94-ca-trust-extracted\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:40.962890 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.962848    5072 generic.go:356] "Generic (PLEG): container finished" podID="395d5cbd-c077-4578-b377-3034e42a2b94" containerID="833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd" exitCode=0
Oct 30 03:57:40.963356 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.962909    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f" event={"ID":"395d5cbd-c077-4578-b377-3034e42a2b94","Type":"ContainerDied","Data":"833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd"}
Oct 30 03:57:40.963356 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.962936    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f"
Oct 30 03:57:40.963356 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.962995    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-759fbb47f9-s2s2f" event={"ID":"395d5cbd-c077-4578-b377-3034e42a2b94","Type":"ContainerDied","Data":"9d33f03496825199ccb3483b58630616822b3e3acdcd451cd3677f63888a8ee4"}
Oct 30 03:57:40.963356 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.963018    5072 scope.go:122] "RemoveContainer" containerID="833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd"
Oct 30 03:57:40.975123 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.975099    5072 scope.go:122] "RemoveContainer" containerID="833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd"
Oct 30 03:57:40.975605 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:40.975575    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd\": container with ID starting with 833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd not found: ID does not exist" containerID="833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd"
Oct 30 03:57:40.975712 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:40.975618    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd"} err="failed to get container status \"833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd\": rpc error: code = NotFound desc = could not find container \"833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd\": container with ID starting with 833d664ebcde5b2c17622d1d99b60af2c1b470813de738294a6f1a393b5fffdd not found: ID does not exist"
Oct 30 03:57:41.037176 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:41.037138    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-759fbb47f9-s2s2f"]
Oct 30 03:57:41.043320 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:41.043283    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-759fbb47f9-s2s2f"]
Oct 30 03:57:41.742802 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:41.741489    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:41.762099 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:57:41.761915    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 03:57:41.763404 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:41.763378    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:41.992630 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:41.992601    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:57:42.297228 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:42.297192    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:57:42.412465 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:42.411102    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="395d5cbd-c077-4578-b377-3034e42a2b94" path="/var/lib/kubelet/pods/395d5cbd-c077-4578-b377-3034e42a2b94/volumes"
Oct 30 03:57:45.145435 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:45.145399    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:45.145975 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:45.145471    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:45.151354 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:45.151329    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:46.011109 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:46.011064    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:57:46.086901 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:46.086861    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8565dfb875-f75g8"]
Oct 30 03:57:47.602921 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:47.602884    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:47.603301 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:47.602971    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:57:47.663724 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:47.663689    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-qddvs-q47fm"]
Oct 30 03:57:47.663989 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:47.663961    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-qddvs-q47fm" podUID="67dad9d6-9671-49f3-84a2-024463451062" containerName="insights-gathering" containerID="cri-o://e97abdeeb6ea2d5d5c7f770caf73c93d7fcd2711bf71f3f4d25ae5460d6b71f8" gracePeriod=30
Oct 30 03:57:48.019384 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.019353    5072 generic.go:356] "Generic (PLEG): container finished" podID="67dad9d6-9671-49f3-84a2-024463451062" containerID="e97abdeeb6ea2d5d5c7f770caf73c93d7fcd2711bf71f3f4d25ae5460d6b71f8" exitCode=2
Oct 30 03:57:48.019544 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.019426    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qddvs-q47fm" event={"ID":"67dad9d6-9671-49f3-84a2-024463451062","Type":"ContainerDied","Data":"e97abdeeb6ea2d5d5c7f770caf73c93d7fcd2711bf71f3f4d25ae5460d6b71f8"}
Oct 30 03:57:48.019544 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.019492    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qddvs-q47fm" event={"ID":"67dad9d6-9671-49f3-84a2-024463451062","Type":"ContainerDied","Data":"d0353988f43af1839855daa5855f75647988e0f6163e7790e44cd475190d00eb"}
Oct 30 03:57:48.019544 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.019514    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0353988f43af1839855daa5855f75647988e0f6163e7790e44cd475190d00eb"
Oct 30 03:57:48.025313 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.025289    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:57:48.119571 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.119535    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67dad9d6-9671-49f3-84a2-024463451062-kube-api-access-xq4h9\" (UniqueName: \"kubernetes.io/projected/67dad9d6-9671-49f3-84a2-024463451062-kube-api-access-xq4h9\") pod \"67dad9d6-9671-49f3-84a2-024463451062\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") "
Oct 30 03:57:48.119726 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.119636    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-trusted-ca-bundle\") pod \"67dad9d6-9671-49f3-84a2-024463451062\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") "
Oct 30 03:57:48.120413 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.120381    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-trusted-ca-bundle" pod "67dad9d6-9671-49f3-84a2-024463451062" (UID: "67dad9d6-9671-49f3-84a2-024463451062"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:48.122447 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.122414    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dad9d6-9671-49f3-84a2-024463451062-kube-api-access-xq4h9" pod "67dad9d6-9671-49f3-84a2-024463451062" (UID: "67dad9d6-9671-49f3-84a2-024463451062"). InnerVolumeSpecName "kube-api-access-xq4h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:48.220404 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.220373    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dad9d6-9671-49f3-84a2-024463451062-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/67dad9d6-9671-49f3-84a2-024463451062-archives-path\") pod \"67dad9d6-9671-49f3-84a2-024463451062\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") "
Oct 30 03:57:48.220539 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.220411    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-service-ca-bundle\") pod \"67dad9d6-9671-49f3-84a2-024463451062\" (UID: \"67dad9d6-9671-49f3-84a2-024463451062\") "
Oct 30 03:57:48.220697 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.220675    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dad9d6-9671-49f3-84a2-024463451062-archives-path" pod "67dad9d6-9671-49f3-84a2-024463451062" (UID: "67dad9d6-9671-49f3-84a2-024463451062"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:48.220734 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.220686    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq4h9\" (UniqueName: \"kubernetes.io/projected/67dad9d6-9671-49f3-84a2-024463451062-kube-api-access-xq4h9\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:48.220734 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.220728    5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-trusted-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:48.220798 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.220728    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-service-ca-bundle" pod "67dad9d6-9671-49f3-84a2-024463451062" (UID: "67dad9d6-9671-49f3-84a2-024463451062"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:48.321296 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.321267    5072 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/67dad9d6-9671-49f3-84a2-024463451062-archives-path\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:48.321296 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:48.321294    5072 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67dad9d6-9671-49f3-84a2-024463451062-service-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:49.025274 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:49.025246    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qddvs-q47fm"
Oct 30 03:57:49.080487 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:49.080448    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-qddvs-q47fm"]
Oct 30 03:57:49.093369 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:49.093344    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-qddvs-q47fm"]
Oct 30 03:57:50.407047 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:50.407008    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67dad9d6-9671-49f3-84a2-024463451062" path="/var/lib/kubelet/pods/67dad9d6-9671-49f3-84a2-024463451062/volumes"
Oct 30 03:57:54.928271 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:54.928198    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-85c4fcd595-5xn9h" podUID="63ac0921-404c-493d-9aec-02c93599b494" containerName="console" containerID="cri-o://fab5b5292926f9e51a6d4f52ff40d8f346edb5f1dd7e29143493feaa7fa09802" gracePeriod=15
Oct 30 03:57:55.057285 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.057251    5072 generic.go:356] "Generic (PLEG): container finished" podID="63ac0921-404c-493d-9aec-02c93599b494" containerID="fab5b5292926f9e51a6d4f52ff40d8f346edb5f1dd7e29143493feaa7fa09802" exitCode=0
Oct 30 03:57:55.057428 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.057324    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c4fcd595-5xn9h" event={"ID":"63ac0921-404c-493d-9aec-02c93599b494","Type":"ContainerDied","Data":"fab5b5292926f9e51a6d4f52ff40d8f346edb5f1dd7e29143493feaa7fa09802"}
Oct 30 03:57:55.273179 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.273149    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:55.284997 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.284975    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ac0921-404c-493d-9aec-02c93599b494-tmp\" (UniqueName: \"kubernetes.io/empty-dir/63ac0921-404c-493d-9aec-02c93599b494-tmp\") pod \"63ac0921-404c-493d-9aec-02c93599b494\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") "
Oct 30 03:57:55.285145 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285014    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ac0921-404c-493d-9aec-02c93599b494-kube-api-access-zspg8\" (UniqueName: \"kubernetes.io/projected/63ac0921-404c-493d-9aec-02c93599b494-kube-api-access-zspg8\") pod \"63ac0921-404c-493d-9aec-02c93599b494\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") "
Oct 30 03:57:55.285145 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285064    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-console-config\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-console-config\") pod \"63ac0921-404c-493d-9aec-02c93599b494\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") "
Oct 30 03:57:55.285145 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285121    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-oauth-serving-cert\") pod \"63ac0921-404c-493d-9aec-02c93599b494\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") "
Oct 30 03:57:55.285323 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285167    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-oauth-config\") pod \"63ac0921-404c-493d-9aec-02c93599b494\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") "
Oct 30 03:57:55.285323 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285199    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-serving-cert\") pod \"63ac0921-404c-493d-9aec-02c93599b494\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") "
Oct 30 03:57:55.285323 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285280    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ac0921-404c-493d-9aec-02c93599b494-tmp" pod "63ac0921-404c-493d-9aec-02c93599b494" (UID: "63ac0921-404c-493d-9aec-02c93599b494"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:55.285323 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285310    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-service-ca\") pod \"63ac0921-404c-493d-9aec-02c93599b494\" (UID: \"63ac0921-404c-493d-9aec-02c93599b494\") "
Oct 30 03:57:55.285528 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285481    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-console-config" pod "63ac0921-404c-493d-9aec-02c93599b494" (UID: "63ac0921-404c-493d-9aec-02c93599b494"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:55.285587 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285554    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-oauth-serving-cert" pod "63ac0921-404c-493d-9aec-02c93599b494" (UID: "63ac0921-404c-493d-9aec-02c93599b494"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:55.285735 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285717    5072 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63ac0921-404c-493d-9aec-02c93599b494-tmp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:55.285735 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285742    5072 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-console-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:55.285840 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285758    5072 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-oauth-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:55.285840 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.285775    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-service-ca" pod "63ac0921-404c-493d-9aec-02c93599b494" (UID: "63ac0921-404c-493d-9aec-02c93599b494"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:55.287625 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.287606    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ac0921-404c-493d-9aec-02c93599b494-kube-api-access-zspg8" pod "63ac0921-404c-493d-9aec-02c93599b494" (UID: "63ac0921-404c-493d-9aec-02c93599b494"). InnerVolumeSpecName "kube-api-access-zspg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:55.288003 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.287979    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-oauth-config" pod "63ac0921-404c-493d-9aec-02c93599b494" (UID: "63ac0921-404c-493d-9aec-02c93599b494"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:55.288071 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.287998    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-serving-cert" pod "63ac0921-404c-493d-9aec-02c93599b494" (UID: "63ac0921-404c-493d-9aec-02c93599b494"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:55.387017 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.386988    5072 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63ac0921-404c-493d-9aec-02c93599b494-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:55.387017 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.387013    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zspg8\" (UniqueName: \"kubernetes.io/projected/63ac0921-404c-493d-9aec-02c93599b494-kube-api-access-zspg8\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:55.387017 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.387024    5072 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-oauth-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:55.387265 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:55.387033    5072 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63ac0921-404c-493d-9aec-02c93599b494-console-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:57:56.064377 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:56.064344    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c4fcd595-5xn9h" event={"ID":"63ac0921-404c-493d-9aec-02c93599b494","Type":"ContainerDied","Data":"f798d77e62b0f336d1ec61dab05c1fe54e84fc76bae7edf583ab5acb2e8fad68"}
Oct 30 03:57:56.064799 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:56.064386    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c4fcd595-5xn9h"
Oct 30 03:57:56.064799 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:56.064401    5072 scope.go:122] "RemoveContainer" containerID="fab5b5292926f9e51a6d4f52ff40d8f346edb5f1dd7e29143493feaa7fa09802"
Oct 30 03:57:56.117468 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:56.117443    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c4fcd595-5xn9h"]
Oct 30 03:57:56.135647 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:56.131636    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85c4fcd595-5xn9h"]
Oct 30 03:57:56.406938 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:56.406845    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ac0921-404c-493d-9aec-02c93599b494" path="/var/lib/kubelet/pods/63ac0921-404c-493d-9aec-02c93599b494/volumes"
Oct 30 03:57:59.861566 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:57:59.861501    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-75c989c798-7tc4c" podUID="fe0bb808-beaa-4907-94e0-29b68fe8d121" containerName="console" containerID="cri-o://86c475a381ec3f294c4dbf871c898d9e6b8e90cd2ed7f28d345a1d61a7446c28" gracePeriod=15
Oct 30 03:58:00.089314 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.089276    5072 generic.go:356] "Generic (PLEG): container finished" podID="fe0bb808-beaa-4907-94e0-29b68fe8d121" containerID="86c475a381ec3f294c4dbf871c898d9e6b8e90cd2ed7f28d345a1d61a7446c28" exitCode=0
Oct 30 03:58:00.089478 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.089355    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c989c798-7tc4c" event={"ID":"fe0bb808-beaa-4907-94e0-29b68fe8d121","Type":"ContainerDied","Data":"86c475a381ec3f294c4dbf871c898d9e6b8e90cd2ed7f28d345a1d61a7446c28"}
Oct 30 03:58:00.210810 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.210784    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:58:00.336851 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.336824    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-oauth-serving-cert\") pod \"fe0bb808-beaa-4907-94e0-29b68fe8d121\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") "
Oct 30 03:58:00.336851 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.336853    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-trusted-ca-bundle\") pod \"fe0bb808-beaa-4907-94e0-29b68fe8d121\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") "
Oct 30 03:58:00.337043 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.336919    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-config\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-config\") pod \"fe0bb808-beaa-4907-94e0-29b68fe8d121\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") "
Oct 30 03:58:00.337043 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.336946    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0bb808-beaa-4907-94e0-29b68fe8d121-kube-api-access-x4n6l\" (UniqueName: \"kubernetes.io/projected/fe0bb808-beaa-4907-94e0-29b68fe8d121-kube-api-access-x4n6l\") pod \"fe0bb808-beaa-4907-94e0-29b68fe8d121\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") "
Oct 30 03:58:00.337043 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337028    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0bb808-beaa-4907-94e0-29b68fe8d121-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe0bb808-beaa-4907-94e0-29b68fe8d121-tmp\") pod \"fe0bb808-beaa-4907-94e0-29b68fe8d121\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") "
Oct 30 03:58:00.337204 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337119    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-serving-cert\") pod \"fe0bb808-beaa-4907-94e0-29b68fe8d121\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") "
Oct 30 03:58:00.337204 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337147    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-oauth-config\") pod \"fe0bb808-beaa-4907-94e0-29b68fe8d121\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") "
Oct 30 03:58:00.337204 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337182    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-service-ca\") pod \"fe0bb808-beaa-4907-94e0-29b68fe8d121\" (UID: \"fe0bb808-beaa-4907-94e0-29b68fe8d121\") "
Oct 30 03:58:00.337352 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337293    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-oauth-serving-cert" pod "fe0bb808-beaa-4907-94e0-29b68fe8d121" (UID: "fe0bb808-beaa-4907-94e0-29b68fe8d121"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:00.337495 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337461    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0bb808-beaa-4907-94e0-29b68fe8d121-tmp" pod "fe0bb808-beaa-4907-94e0-29b68fe8d121" (UID: "fe0bb808-beaa-4907-94e0-29b68fe8d121"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:00.337495 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337488    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-config" pod "fe0bb808-beaa-4907-94e0-29b68fe8d121" (UID: "fe0bb808-beaa-4907-94e0-29b68fe8d121"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:00.337656 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337502    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-trusted-ca-bundle" pod "fe0bb808-beaa-4907-94e0-29b68fe8d121" (UID: "fe0bb808-beaa-4907-94e0-29b68fe8d121"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:00.337656 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337587    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-service-ca" pod "fe0bb808-beaa-4907-94e0-29b68fe8d121" (UID: "fe0bb808-beaa-4907-94e0-29b68fe8d121"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:00.337764 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337746    5072 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-oauth-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:00.337832 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337772    5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-trusted-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:00.337832 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337789    5072 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:00.337832 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337801    5072 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe0bb808-beaa-4907-94e0-29b68fe8d121-tmp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:00.337832 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.337809    5072 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0bb808-beaa-4907-94e0-29b68fe8d121-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:00.339386 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.339359    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0bb808-beaa-4907-94e0-29b68fe8d121-kube-api-access-x4n6l" pod "fe0bb808-beaa-4907-94e0-29b68fe8d121" (UID: "fe0bb808-beaa-4907-94e0-29b68fe8d121"). InnerVolumeSpecName "kube-api-access-x4n6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:00.339530 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.339505    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-oauth-config" pod "fe0bb808-beaa-4907-94e0-29b68fe8d121" (UID: "fe0bb808-beaa-4907-94e0-29b68fe8d121"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:00.339592 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.339563    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-serving-cert" pod "fe0bb808-beaa-4907-94e0-29b68fe8d121" (UID: "fe0bb808-beaa-4907-94e0-29b68fe8d121"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:00.438653 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.438632    5072 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:00.438653 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.438654    5072 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe0bb808-beaa-4907-94e0-29b68fe8d121-console-oauth-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:00.438793 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:00.438665    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4n6l\" (UniqueName: \"kubernetes.io/projected/fe0bb808-beaa-4907-94e0-29b68fe8d121-kube-api-access-x4n6l\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:01.096205 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:01.096176    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c989c798-7tc4c"
Oct 30 03:58:01.096205 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:01.096196    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c989c798-7tc4c" event={"ID":"fe0bb808-beaa-4907-94e0-29b68fe8d121","Type":"ContainerDied","Data":"2bd04fade11615a37e724d4e01c7fd9093d3a3e1af655dc800cd0b80d921f4c0"}
Oct 30 03:58:01.096716 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:01.096259    5072 scope.go:122] "RemoveContainer" containerID="86c475a381ec3f294c4dbf871c898d9e6b8e90cd2ed7f28d345a1d61a7446c28"
Oct 30 03:58:01.149444 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:01.149409    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c989c798-7tc4c"]
Oct 30 03:58:01.163491 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:01.163463    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75c989c798-7tc4c"]
Oct 30 03:58:02.408030 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:02.408000    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0bb808-beaa-4907-94e0-29b68fe8d121" path="/var/lib/kubelet/pods/fe0bb808-beaa-4907-94e0-29b68fe8d121/volumes"
Oct 30 03:58:02.734780 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:02.734708    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf" podUID="343419d3-210d-4c12-86ff-c92f8f574b27" containerName="registry" containerID="cri-o://4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3" gracePeriod=30
Oct 30 03:58:03.038735 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.038707    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:58:03.108810 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.108777    5072 generic.go:356] "Generic (PLEG): container finished" podID="343419d3-210d-4c12-86ff-c92f8f574b27" containerID="4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3" exitCode=0
Oct 30 03:58:03.109003 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.108831    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf"
Oct 30 03:58:03.109003 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.108875    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf" event={"ID":"343419d3-210d-4c12-86ff-c92f8f574b27","Type":"ContainerDied","Data":"4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3"}
Oct 30 03:58:03.109003 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.108968    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bbcfd588b-rlkpf" event={"ID":"343419d3-210d-4c12-86ff-c92f8f574b27","Type":"ContainerDied","Data":"7e5d7902ec58915cda31f78ed7be07838036933f99c510215cf82465dea8cfe0"}
Oct 30 03:58:03.109003 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.108999    5072 scope.go:122] "RemoveContainer" containerID="4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3"
Oct 30 03:58:03.118513 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.118496    5072 scope.go:122] "RemoveContainer" containerID="4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3"
Oct 30 03:58:03.118799 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:58:03.118776    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3\": container with ID starting with 4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3 not found: ID does not exist" containerID="4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3"
Oct 30 03:58:03.118895 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.118809    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3"} err="failed to get container status \"4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3\": rpc error: code = NotFound desc = could not find container \"4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3\": container with ID starting with 4344dfa5d28ec8b1ae0db33a98782aef0d4667f7e313e2588346521cb1d965d3 not found: ID does not exist"
Oct 30 03:58:03.164642 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.164615    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-trusted-ca\") pod \"343419d3-210d-4c12-86ff-c92f8f574b27\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") "
Oct 30 03:58:03.164768 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.164656    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-registry-certificates\") pod \"343419d3-210d-4c12-86ff-c92f8f574b27\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") "
Oct 30 03:58:03.164768 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.164714    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343419d3-210d-4c12-86ff-c92f8f574b27-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/343419d3-210d-4c12-86ff-c92f8f574b27-ca-trust-extracted\") pod \"343419d3-210d-4c12-86ff-c92f8f574b27\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") "
Oct 30 03:58:03.164768 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.164737    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-image-registry-private-configuration\") pod \"343419d3-210d-4c12-86ff-c92f8f574b27\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") "
Oct 30 03:58:03.164959 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.164771    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-kube-api-access-vzdlq\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-kube-api-access-vzdlq\") pod \"343419d3-210d-4c12-86ff-c92f8f574b27\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") "
Oct 30 03:58:03.165014 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.164974    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") pod \"343419d3-210d-4c12-86ff-c92f8f574b27\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") "
Oct 30 03:58:03.165066 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.165033    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-bound-sa-token\") pod \"343419d3-210d-4c12-86ff-c92f8f574b27\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") "
Oct 30 03:58:03.165163 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.165095    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-trusted-ca" pod "343419d3-210d-4c12-86ff-c92f8f574b27" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:03.165163 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.165134    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-registry-certificates" pod "343419d3-210d-4c12-86ff-c92f8f574b27" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:03.165163 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.165155    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-installation-pull-secrets\") pod \"343419d3-210d-4c12-86ff-c92f8f574b27\" (UID: \"343419d3-210d-4c12-86ff-c92f8f574b27\") "
Oct 30 03:58:03.165547 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.165522    5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-trusted-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:03.165671 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.165554    5072 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/343419d3-210d-4c12-86ff-c92f8f574b27-registry-certificates\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:03.167434 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.167401    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-kube-api-access-vzdlq" pod "343419d3-210d-4c12-86ff-c92f8f574b27" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27"). InnerVolumeSpecName "kube-api-access-vzdlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:03.167646 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.167623    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-bound-sa-token" pod "343419d3-210d-4c12-86ff-c92f8f574b27" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:03.167956 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.167928    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls" pod "343419d3-210d-4c12-86ff-c92f8f574b27" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:03.168032 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.167954    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-installation-pull-secrets" pod "343419d3-210d-4c12-86ff-c92f8f574b27" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:03.168331 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.168306    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-image-registry-private-configuration" pod "343419d3-210d-4c12-86ff-c92f8f574b27" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:03.173465 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.173433    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343419d3-210d-4c12-86ff-c92f8f574b27-ca-trust-extracted" pod "343419d3-210d-4c12-86ff-c92f8f574b27" (UID: "343419d3-210d-4c12-86ff-c92f8f574b27"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:03.266921 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.266866    5072 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/343419d3-210d-4c12-86ff-c92f8f574b27-ca-trust-extracted\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:03.266921 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.266889    5072 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-image-registry-private-configuration\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:03.266921 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.266899    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzdlq\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-kube-api-access-vzdlq\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:03.266921 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.266909    5072 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-registry-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:03.266921 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.266918    5072 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/343419d3-210d-4c12-86ff-c92f8f574b27-bound-sa-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:03.267163 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.266926    5072 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/343419d3-210d-4c12-86ff-c92f8f574b27-installation-pull-secrets\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:03.455735 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.455711    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bbcfd588b-rlkpf"]
Oct 30 03:58:03.468203 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:03.468180    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bbcfd588b-rlkpf"]
Oct 30 03:58:04.407595 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:04.407567    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="343419d3-210d-4c12-86ff-c92f8f574b27" path="/var/lib/kubelet/pods/343419d3-210d-4c12-86ff-c92f8f574b27/volumes"
Oct 30 03:58:07.609197 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:07.609171    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:58:07.612800 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:07.612781    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7858fcf8fc-qc6gr"
Oct 30 03:58:11.121506 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.121457    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8565dfb875-f75g8" podUID="716a3fab-e121-4291-8ef5-02a47719ccbb" containerName="console" containerID="cri-o://81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c" gracePeriod=15
Oct 30 03:58:11.451382 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.451355    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:58:11.540358 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.540330    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-trusted-ca-bundle\") pod \"716a3fab-e121-4291-8ef5-02a47719ccbb\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") "
Oct 30 03:58:11.540583 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.540384    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-serving-cert\") pod \"716a3fab-e121-4291-8ef5-02a47719ccbb\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") "
Oct 30 03:58:11.540583 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.540408    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-oauth-config\") pod \"716a3fab-e121-4291-8ef5-02a47719ccbb\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") "
Oct 30 03:58:11.540696 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.540578    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-console-config\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-console-config\") pod \"716a3fab-e121-4291-8ef5-02a47719ccbb\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") "
Oct 30 03:58:11.540696 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.540634    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-oauth-serving-cert\") pod \"716a3fab-e121-4291-8ef5-02a47719ccbb\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") "
Oct 30 03:58:11.540805 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.540724    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-service-ca\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-service-ca\") pod \"716a3fab-e121-4291-8ef5-02a47719ccbb\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") "
Oct 30 03:58:11.540805 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.540729    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-trusted-ca-bundle" pod "716a3fab-e121-4291-8ef5-02a47719ccbb" (UID: "716a3fab-e121-4291-8ef5-02a47719ccbb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:11.540805 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.540794    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716a3fab-e121-4291-8ef5-02a47719ccbb-kube-api-access-w9ppn\" (UniqueName: \"kubernetes.io/projected/716a3fab-e121-4291-8ef5-02a47719ccbb-kube-api-access-w9ppn\") pod \"716a3fab-e121-4291-8ef5-02a47719ccbb\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") "
Oct 30 03:58:11.540961 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.540828    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/716a3fab-e121-4291-8ef5-02a47719ccbb-tmp\" (UniqueName: \"kubernetes.io/empty-dir/716a3fab-e121-4291-8ef5-02a47719ccbb-tmp\") pod \"716a3fab-e121-4291-8ef5-02a47719ccbb\" (UID: \"716a3fab-e121-4291-8ef5-02a47719ccbb\") "
Oct 30 03:58:11.541059 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.541032    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-oauth-serving-cert" pod "716a3fab-e121-4291-8ef5-02a47719ccbb" (UID: "716a3fab-e121-4291-8ef5-02a47719ccbb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:11.541157 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.541060    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-console-config" pod "716a3fab-e121-4291-8ef5-02a47719ccbb" (UID: "716a3fab-e121-4291-8ef5-02a47719ccbb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:11.541292 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.541255    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716a3fab-e121-4291-8ef5-02a47719ccbb-tmp" pod "716a3fab-e121-4291-8ef5-02a47719ccbb" (UID: "716a3fab-e121-4291-8ef5-02a47719ccbb"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:11.541437 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.541349    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-service-ca" pod "716a3fab-e121-4291-8ef5-02a47719ccbb" (UID: "716a3fab-e121-4291-8ef5-02a47719ccbb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:11.541514 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.541489    5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-trusted-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:11.541575 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.541512    5072 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-console-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:11.541575 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.541529    5072 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-oauth-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:11.542714 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.542694    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-oauth-config" pod "716a3fab-e121-4291-8ef5-02a47719ccbb" (UID: "716a3fab-e121-4291-8ef5-02a47719ccbb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:11.542960 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.542940    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716a3fab-e121-4291-8ef5-02a47719ccbb-kube-api-access-w9ppn" pod "716a3fab-e121-4291-8ef5-02a47719ccbb" (UID: "716a3fab-e121-4291-8ef5-02a47719ccbb"). InnerVolumeSpecName "kube-api-access-w9ppn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:11.543042 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.543004    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-serving-cert" pod "716a3fab-e121-4291-8ef5-02a47719ccbb" (UID: "716a3fab-e121-4291-8ef5-02a47719ccbb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:11.642733 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.642709    5072 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:11.642733 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.642734    5072 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/716a3fab-e121-4291-8ef5-02a47719ccbb-console-oauth-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:11.642896 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.642744    5072 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/716a3fab-e121-4291-8ef5-02a47719ccbb-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:11.642896 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.642752    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9ppn\" (UniqueName: \"kubernetes.io/projected/716a3fab-e121-4291-8ef5-02a47719ccbb-kube-api-access-w9ppn\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:11.642896 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:11.642761    5072 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/716a3fab-e121-4291-8ef5-02a47719ccbb-tmp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:11.753104 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:58:11.753052    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 03:58:12.155253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:12.155165    5072 generic.go:356] "Generic (PLEG): container finished" podID="716a3fab-e121-4291-8ef5-02a47719ccbb" containerID="81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c" exitCode=0
Oct 30 03:58:12.155253 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:12.155241    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8565dfb875-f75g8"
Oct 30 03:58:12.155746 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:12.155267    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8565dfb875-f75g8" event={"ID":"716a3fab-e121-4291-8ef5-02a47719ccbb","Type":"ContainerDied","Data":"81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c"}
Oct 30 03:58:12.155746 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:12.155335    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8565dfb875-f75g8" event={"ID":"716a3fab-e121-4291-8ef5-02a47719ccbb","Type":"ContainerDied","Data":"9976bcf354c95de1797f4f369c9d93a396826849657f72b143371869fcf7f81e"}
Oct 30 03:58:12.155746 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:12.155358    5072 scope.go:122] "RemoveContainer" containerID="81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c"
Oct 30 03:58:12.165465 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:12.165447    5072 scope.go:122] "RemoveContainer" containerID="81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c"
Oct 30 03:58:12.165725 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:58:12.165705    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c\": container with ID starting with 81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c not found: ID does not exist" containerID="81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c"
Oct 30 03:58:12.165772 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:12.165739    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c"} err="failed to get container status \"81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c\": rpc error: code = NotFound desc = could not find container \"81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c\": container with ID starting with 81a7ec0297451348b4fcc62532dd9e6c5c05daf4a8a6a2b2c7d78b7de2b3740c not found: ID does not exist"
Oct 30 03:58:12.208990 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:12.208958    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8565dfb875-f75g8"]
Oct 30 03:58:12.211606 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:12.211583    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8565dfb875-f75g8"]
Oct 30 03:58:12.409148 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:12.408264    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="716a3fab-e121-4291-8ef5-02a47719ccbb" path="/var/lib/kubelet/pods/716a3fab-e121-4291-8ef5-02a47719ccbb/volumes"
Oct 30 03:58:13.672454 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:13.672412    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 03:58:13.672849 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:13.672818    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="alertmanager" containerID="cri-o://9ffedb59c6cff95a4c7d6c93797e5b9d8f90e63e8b784407d7f80c48bee4d296" gracePeriod=120
Oct 30 03:58:13.672920 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:13.672865    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="kube-rbac-proxy-metric" containerID="cri-o://91390c129310e4a3962d846caf83e4ea0984faf224cc92db1d9e3c2ea7287bdb" gracePeriod=120
Oct 30 03:58:13.672976 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:13.672893    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="kube-rbac-proxy-web" containerID="cri-o://85d56d9fdcf727fbf293ab4d01151630c986889a5b1b3b6fd381db6dd532ed5d" gracePeriod=120
Oct 30 03:58:13.672976 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:13.672920    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="config-reloader" containerID="cri-o://65118ece72d44bb45ae90e8d701aa7168cccdfb385ce75bc691e0ff655146a84" gracePeriod=120
Oct 30 03:58:13.673183 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:13.673002    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="prom-label-proxy" containerID="cri-o://7d682b87dc3fb7441e82b4bf0746df95b44564ec82aa09f48ef2dae143ef6a42" gracePeriod=120
Oct 30 03:58:13.673183 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:13.673011    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="kube-rbac-proxy" containerID="cri-o://51a9cd58223c187e09f26eaf662519381e049886ed152e06645a799da125f844" gracePeriod=120
Oct 30 03:58:14.170428 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:14.170399    5072 generic.go:356] "Generic (PLEG): container finished" podID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerID="7d682b87dc3fb7441e82b4bf0746df95b44564ec82aa09f48ef2dae143ef6a42" exitCode=0
Oct 30 03:58:14.170428 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:14.170423    5072 generic.go:356] "Generic (PLEG): container finished" podID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerID="51a9cd58223c187e09f26eaf662519381e049886ed152e06645a799da125f844" exitCode=0
Oct 30 03:58:14.170428 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:14.170429    5072 generic.go:356] "Generic (PLEG): container finished" podID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerID="65118ece72d44bb45ae90e8d701aa7168cccdfb385ce75bc691e0ff655146a84" exitCode=0
Oct 30 03:58:14.170428 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:14.170436    5072 generic.go:356] "Generic (PLEG): container finished" podID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerID="9ffedb59c6cff95a4c7d6c93797e5b9d8f90e63e8b784407d7f80c48bee4d296" exitCode=0
Oct 30 03:58:14.170678 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:14.170473    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerDied","Data":"7d682b87dc3fb7441e82b4bf0746df95b44564ec82aa09f48ef2dae143ef6a42"}
Oct 30 03:58:14.170678 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:14.170540    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerDied","Data":"51a9cd58223c187e09f26eaf662519381e049886ed152e06645a799da125f844"}
Oct 30 03:58:14.170678 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:14.170568    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerDied","Data":"65118ece72d44bb45ae90e8d701aa7168cccdfb385ce75bc691e0ff655146a84"}
Oct 30 03:58:14.170678 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:14.170593    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerDied","Data":"9ffedb59c6cff95a4c7d6c93797e5b9d8f90e63e8b784407d7f80c48bee4d296"}
Oct 30 03:58:15.179915 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.179883    5072 generic.go:356] "Generic (PLEG): container finished" podID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerID="91390c129310e4a3962d846caf83e4ea0984faf224cc92db1d9e3c2ea7287bdb" exitCode=0
Oct 30 03:58:15.179915 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.179914    5072 generic.go:356] "Generic (PLEG): container finished" podID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerID="85d56d9fdcf727fbf293ab4d01151630c986889a5b1b3b6fd381db6dd532ed5d" exitCode=0
Oct 30 03:58:15.180401 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.179955    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerDied","Data":"91390c129310e4a3962d846caf83e4ea0984faf224cc92db1d9e3c2ea7287bdb"}
Oct 30 03:58:15.180401 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.180019    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerDied","Data":"85d56d9fdcf727fbf293ab4d01151630c986889a5b1b3b6fd381db6dd532ed5d"}
Oct 30 03:58:15.181500 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.181480    5072 generic.go:356] "Generic (PLEG): container finished" podID="72c9fb60-7d7b-46b7-8d02-0b51dd1d0221" containerID="0dbc6894e356c47ef9edfea13013effc603875c6104687b31a0dd0e9a583963e" exitCode=0
Oct 30 03:58:15.181568 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.181538    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d" event={"ID":"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221","Type":"ContainerDied","Data":"0dbc6894e356c47ef9edfea13013effc603875c6104687b31a0dd0e9a583963e"}
Oct 30 03:58:15.181837 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.181824    5072 scope.go:122] "RemoveContainer" containerID="0dbc6894e356c47ef9edfea13013effc603875c6104687b31a0dd0e9a583963e"
Oct 30 03:58:15.476238 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.476213    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:15.579616 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.579543    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-metrics-client-ca\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.579616 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.579593    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-out\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.579824 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.579668    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-main-tls\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.579824 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.579697    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-main-db\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.579824 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.579735    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-kube-api-access-p4pgj\" (UniqueName: \"kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-kube-api-access-p4pgj\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.579824 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.579766    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.579824 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.579795    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.580105 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.579823    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-volume\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-volume\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.580105 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.579859    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-tls-assets\" (UniqueName: \"kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-tls-assets\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.580105 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.579892    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-web-config\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-web-config\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.580105 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.579903    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-metrics-client-ca" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:15.580652 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.580372    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-main-db" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:15.580933 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.580908    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-cluster-tls-config\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.581038 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.581017    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-web\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.581133 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.581070    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-trusted-ca-bundle\") pod \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\" (UID: \"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7\") "
Oct 30 03:58:15.581560 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.581535    5072 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-metrics-client-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:15.581639 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.581559    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-trusted-ca-bundle" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:15.581639 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.581566    5072 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-main-db\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:15.589758 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.589424    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-volume" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:15.589758 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.589474    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-kube-api-access-p4pgj" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "kube-api-access-p4pgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:15.589758 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.589562    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-tls-assets" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:15.589758 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.589625    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-main-tls" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:15.589758 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.589643    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:15.589758 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.589659    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-metric" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:15.589758 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.589713    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-out" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:15.591561 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.591525    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-cluster-tls-config" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:15.592315 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.592261    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-web" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:15.603339 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.603311    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-web-config" pod "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" (UID: "497cda50-9fcd-4e3b-b7ab-93d9b599f9a7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:15.683022 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.682996    5072 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-cluster-tls-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:15.683022 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.683021    5072 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:15.683185 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.683031    5072 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:15.683185 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.683040    5072 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-out\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:15.683185 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.683048    5072 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-main-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:15.683185 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.683056    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4pgj\" (UniqueName: \"kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-kube-api-access-p4pgj\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:15.683185 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.683070    5072 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:15.683185 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.683097    5072 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:15.683185 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.683105    5072 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:15.683185 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.683115    5072 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-tls-assets\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:15.683185 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:15.683122    5072 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7-web-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:16.188380 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.188340    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b4n4d" event={"ID":"72c9fb60-7d7b-46b7-8d02-0b51dd1d0221","Type":"ContainerStarted","Data":"98a19758e0c4cd1ce496b8919d83e1ad0f818a2f05638c5773985f7cf16b64f3"}
Oct 30 03:58:16.191571 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.191545    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"497cda50-9fcd-4e3b-b7ab-93d9b599f9a7","Type":"ContainerDied","Data":"0eee56902436eaa9696e722c73086b3cde00c5cd9d2cc47da909b1f5457d27a0"}
Oct 30 03:58:16.191690 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.191618    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.191690 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.191628    5072 scope.go:122] "RemoveContainer" containerID="7d682b87dc3fb7441e82b4bf0746df95b44564ec82aa09f48ef2dae143ef6a42"
Oct 30 03:58:16.200527 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.200510    5072 scope.go:122] "RemoveContainer" containerID="91390c129310e4a3962d846caf83e4ea0984faf224cc92db1d9e3c2ea7287bdb"
Oct 30 03:58:16.208063 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.208045    5072 scope.go:122] "RemoveContainer" containerID="51a9cd58223c187e09f26eaf662519381e049886ed152e06645a799da125f844"
Oct 30 03:58:16.215293 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.215275    5072 scope.go:122] "RemoveContainer" containerID="85d56d9fdcf727fbf293ab4d01151630c986889a5b1b3b6fd381db6dd532ed5d"
Oct 30 03:58:16.222562 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.222541    5072 scope.go:122] "RemoveContainer" containerID="65118ece72d44bb45ae90e8d701aa7168cccdfb385ce75bc691e0ff655146a84"
Oct 30 03:58:16.232616 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.232587    5072 scope.go:122] "RemoveContainer" containerID="9ffedb59c6cff95a4c7d6c93797e5b9d8f90e63e8b784407d7f80c48bee4d296"
Oct 30 03:58:16.242887 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.242863    5072 scope.go:122] "RemoveContainer" containerID="38420e8d51a6b021c1f94f1ac6b973d3b0a5a0ae377602280ea1b877c2758199"
Oct 30 03:58:16.274327 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.274307    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 03:58:16.288383 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.288359    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 03:58:16.407047 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:16.407015    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" path="/var/lib/kubelet/pods/497cda50-9fcd-4e3b-b7ab-93d9b599f9a7/volumes"
Oct 30 03:58:18.263400 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:18.263365    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 03:58:18.263939 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:18.263707    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="prometheus" containerID="cri-o://96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5" gracePeriod=600
Oct 30 03:58:18.264179 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:18.264142    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4" gracePeriod=600
Oct 30 03:58:18.264277 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:18.264250    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="kube-rbac-proxy" containerID="cri-o://9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb" gracePeriod=600
Oct 30 03:58:18.264333 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:18.264319    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="kube-rbac-proxy-web" containerID="cri-o://350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464" gracePeriod=600
Oct 30 03:58:18.264389 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:18.264380    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="thanos-sidecar" containerID="cri-o://1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a" gracePeriod=600
Oct 30 03:58:18.264474 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:18.264450    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="config-reloader" containerID="cri-o://d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648" gracePeriod=600
Oct 30 03:58:19.211614 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.211581    5072 generic.go:356] "Generic (PLEG): container finished" podID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerID="2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4" exitCode=0
Oct 30 03:58:19.211614 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.211607    5072 generic.go:356] "Generic (PLEG): container finished" podID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerID="9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb" exitCode=0
Oct 30 03:58:19.211614 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.211614    5072 generic.go:356] "Generic (PLEG): container finished" podID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerID="1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a" exitCode=0
Oct 30 03:58:19.211614 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.211619    5072 generic.go:356] "Generic (PLEG): container finished" podID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerID="d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648" exitCode=0
Oct 30 03:58:19.211873 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.211624    5072 generic.go:356] "Generic (PLEG): container finished" podID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerID="96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5" exitCode=0
Oct 30 03:58:19.211873 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.211648    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerDied","Data":"2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4"}
Oct 30 03:58:19.211873 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.211733    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerDied","Data":"9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb"}
Oct 30 03:58:19.211873 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.211759    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerDied","Data":"1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a"}
Oct 30 03:58:19.211873 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.211782    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerDied","Data":"d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648"}
Oct 30 03:58:19.211873 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.211807    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerDied","Data":"96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5"}
Oct 30 03:58:19.595385 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.595359    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:58:19.721272 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721238    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-kube-api-access-5h7sp\" (UniqueName: \"kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-kube-api-access-5h7sp\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.721448 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721277    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-serving-certs-ca-bundle\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.721448 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721318    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-tls-assets\" (UniqueName: \"kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-tls-assets\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.721448 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721369    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-kubelet-serving-ca-bundle\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.721448 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721398    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-2\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.721448 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721440    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.721709 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721458    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-metrics-client-ca\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.721709 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721495    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-kube-rbac-proxy\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.721709 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721525    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-metrics-client-certs\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.721709 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721551    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-1\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.721709 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721607    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-thanos-prometheus-http-client-file\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.721709 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721645    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-db\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.721709 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721685    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-trusted-ca-bundle\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.722102 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721714    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-config\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-config\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.722102 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721770    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-kubelet-serving-ca-bundle" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:19.722102 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.721807    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-serving-certs-ca-bundle" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:19.722252 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.722187    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-2" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:19.722252 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.722058    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-metrics-client-ca" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:19.722356 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.722278    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-1" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:19.722674 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.722650    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-trusted-ca-bundle" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:19.723528 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.723490    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-db" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:19.723732 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.723714    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-web-config\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-web-config\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.724189 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.724170    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.724632 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.724355    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-tls\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.724834 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.724241    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-kube-api-access-5h7sp" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "kube-api-access-5h7sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:19.724834 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.724568    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-metrics-client-certs" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:19.724834 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.724636    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-thanos-prometheus-http-client-file" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:19.725206 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.724839    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-tls-assets" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:19.725355 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.725334    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-0\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.725490 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.725465    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-config-out\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.725585 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.725550    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-kube-rbac-proxy" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:19.725654 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.725635    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-grpc-tls\") pod \"2bd8b824-8c21-40aa-9a1d-f3913690e556\" (UID: \"2bd8b824-8c21-40aa-9a1d-f3913690e556\") "
Oct 30 03:58:19.726004 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.725981    5072 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-tls-assets\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.726130 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.726008    5072 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.726130 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.726023    5072 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.726130 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.726037    5072 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-metrics-client-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.726130 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.726053    5072 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-kube-rbac-proxy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.726130 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.726068    5072 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-metrics-client-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.726130 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.726111    5072 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.726130 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.726126    5072 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.726497 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.726141    5072 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-db\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.726497 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.726156    5072 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.726497 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.726172    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h7sp\" (UniqueName: \"kubernetes.io/projected/2bd8b824-8c21-40aa-9a1d-f3913690e556-kube-api-access-5h7sp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.726497 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.726186    5072 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.726828 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.726806    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-0" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:19.727915 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.727849    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-config" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:19.728029 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.728010    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-kube-rbac-proxy-web" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:19.728155 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.728099    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-thanos-sidecar-tls" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:19.728246 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.728181    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-config-out" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:19.728518 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.728492    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-tls" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:19.729625 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.729603    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-grpc-tls" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:19.741833 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.741809    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-web-config" pod "2bd8b824-8c21-40aa-9a1d-f3913690e556" (UID: "2bd8b824-8c21-40aa-9a1d-f3913690e556"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:19.827524 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.827467    5072 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.827524 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.827493    5072 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-web-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.827524 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.827505    5072 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.827524 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.827514    5072 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.827524 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.827525    5072 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2bd8b824-8c21-40aa-9a1d-f3913690e556-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.827731 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.827534    5072 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2bd8b824-8c21-40aa-9a1d-f3913690e556-config-out\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.827731 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.827542    5072 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-grpc-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:19.827731 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:19.827549    5072 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2bd8b824-8c21-40aa-9a1d-f3913690e556-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:20.220436 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.220405    5072 generic.go:356] "Generic (PLEG): container finished" podID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerID="350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464" exitCode=0
Oct 30 03:58:20.220626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.220452    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerDied","Data":"350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464"}
Oct 30 03:58:20.220626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.220505    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 03:58:20.220626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.220524    5072 scope.go:122] "RemoveContainer" containerID="2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4"
Oct 30 03:58:20.220626 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.220512    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bd8b824-8c21-40aa-9a1d-f3913690e556","Type":"ContainerDied","Data":"113c98977b1cf937aab0fd80af518edf2c2091682e6adf2e92e6a776060635c7"}
Oct 30 03:58:20.229100 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.229062    5072 scope.go:122] "RemoveContainer" containerID="9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb"
Oct 30 03:58:20.236643 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.236609    5072 scope.go:122] "RemoveContainer" containerID="350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464"
Oct 30 03:58:20.244064 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.244044    5072 scope.go:122] "RemoveContainer" containerID="1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a"
Oct 30 03:58:20.251585 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.251569    5072 scope.go:122] "RemoveContainer" containerID="d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648"
Oct 30 03:58:20.258991 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.258974    5072 scope.go:122] "RemoveContainer" containerID="96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5"
Oct 30 03:58:20.266625 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.266610    5072 scope.go:122] "RemoveContainer" containerID="29276ff4a38cbe9f9cc9c9ab8a05e7b4a3993af4e124a07d73bb9d8bd1b19c42"
Oct 30 03:58:20.280292 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.280270    5072 scope.go:122] "RemoveContainer" containerID="2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4"
Oct 30 03:58:20.280558 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:58:20.280541    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4\": container with ID starting with 2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4 not found: ID does not exist" containerID="2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4"
Oct 30 03:58:20.280608 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.280566    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4"} err="failed to get container status \"2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4\": rpc error: code = NotFound desc = could not find container \"2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4\": container with ID starting with 2801c9cbbef456f6ed43cf031afef7dbd3d011f1b69948d56144b58d7fa5cde4 not found: ID does not exist"
Oct 30 03:58:20.280608 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.280582    5072 scope.go:122] "RemoveContainer" containerID="9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb"
Oct 30 03:58:20.280806 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:58:20.280792    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb\": container with ID starting with 9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb not found: ID does not exist" containerID="9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb"
Oct 30 03:58:20.280848 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.280809    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb"} err="failed to get container status \"9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb\": rpc error: code = NotFound desc = could not find container \"9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb\": container with ID starting with 9e642809c9ee96459887fd4aa521589c4b3cdd5119b92eb2b8aa4068eeea25cb not found: ID does not exist"
Oct 30 03:58:20.280848 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.280821    5072 scope.go:122] "RemoveContainer" containerID="350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464"
Oct 30 03:58:20.281013 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:58:20.280997    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464\": container with ID starting with 350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464 not found: ID does not exist" containerID="350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464"
Oct 30 03:58:20.281059 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.281017    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464"} err="failed to get container status \"350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464\": rpc error: code = NotFound desc = could not find container \"350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464\": container with ID starting with 350c75d7c4dcb9fec8b778e3ce1cf425f7a1a8f18699badbad0cf608ca86c464 not found: ID does not exist"
Oct 30 03:58:20.281059 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.281029    5072 scope.go:122] "RemoveContainer" containerID="1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a"
Oct 30 03:58:20.281269 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:58:20.281251    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a\": container with ID starting with 1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a not found: ID does not exist" containerID="1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a"
Oct 30 03:58:20.281332 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.281278    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a"} err="failed to get container status \"1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a\": rpc error: code = NotFound desc = could not find container \"1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a\": container with ID starting with 1d9f99fc1b2f9e143311596fa3ae9f62da042a16bd794122bfc1a8da1c35b08a not found: ID does not exist"
Oct 30 03:58:20.281332 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.281300    5072 scope.go:122] "RemoveContainer" containerID="d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648"
Oct 30 03:58:20.281536 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:58:20.281519    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648\": container with ID starting with d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648 not found: ID does not exist" containerID="d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648"
Oct 30 03:58:20.281588 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.281541    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648"} err="failed to get container status \"d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648\": rpc error: code = NotFound desc = could not find container \"d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648\": container with ID starting with d97f2e3985169497df54aad9335dd66613e48bb8b37b18858c2c731fa623c648 not found: ID does not exist"
Oct 30 03:58:20.281588 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.281558    5072 scope.go:122] "RemoveContainer" containerID="96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5"
Oct 30 03:58:20.281786 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:58:20.281765    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5\": container with ID starting with 96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5 not found: ID does not exist" containerID="96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5"
Oct 30 03:58:20.281851 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.281794    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5"} err="failed to get container status \"96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5\": rpc error: code = NotFound desc = could not find container \"96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5\": container with ID starting with 96c9cee2d4be6b1896767d57499162731a9e7d180a6825c20f2c8c0ba593b7c5 not found: ID does not exist"
Oct 30 03:58:20.281851 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.281814    5072 scope.go:122] "RemoveContainer" containerID="29276ff4a38cbe9f9cc9c9ab8a05e7b4a3993af4e124a07d73bb9d8bd1b19c42"
Oct 30 03:58:20.282063 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:58:20.282047    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29276ff4a38cbe9f9cc9c9ab8a05e7b4a3993af4e124a07d73bb9d8bd1b19c42\": container with ID starting with 29276ff4a38cbe9f9cc9c9ab8a05e7b4a3993af4e124a07d73bb9d8bd1b19c42 not found: ID does not exist" containerID="29276ff4a38cbe9f9cc9c9ab8a05e7b4a3993af4e124a07d73bb9d8bd1b19c42"
Oct 30 03:58:20.282129 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.282069    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29276ff4a38cbe9f9cc9c9ab8a05e7b4a3993af4e124a07d73bb9d8bd1b19c42"} err="failed to get container status \"29276ff4a38cbe9f9cc9c9ab8a05e7b4a3993af4e124a07d73bb9d8bd1b19c42\": rpc error: code = NotFound desc = could not find container \"29276ff4a38cbe9f9cc9c9ab8a05e7b4a3993af4e124a07d73bb9d8bd1b19c42\": container with ID starting with 29276ff4a38cbe9f9cc9c9ab8a05e7b4a3993af4e124a07d73bb9d8bd1b19c42 not found: ID does not exist"
Oct 30 03:58:20.291745 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.291725    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 03:58:20.304742 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.304723    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 03:58:20.406618 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:20.406589    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" path="/var/lib/kubelet/pods/2bd8b824-8c21-40aa-9a1d-f3913690e556/volumes"
Oct 30 03:58:22.917254 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:22.917225    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-869d9c4d7-49r74"]
Oct 30 03:58:35.298919 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:35.298842    5072 generic.go:356] "Generic (PLEG): container finished" podID="1fe5ff28-0896-4f54-b5cc-5326a0d60752" containerID="50ca62ec3fb587bd22466f206da0132db84f9ea73e44cd802dbd232beadf7577" exitCode=0
Oct 30 03:58:35.299318 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:35.298916    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx" event={"ID":"1fe5ff28-0896-4f54-b5cc-5326a0d60752","Type":"ContainerDied","Data":"50ca62ec3fb587bd22466f206da0132db84f9ea73e44cd802dbd232beadf7577"}
Oct 30 03:58:35.299318 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:35.299233    5072 scope.go:122] "RemoveContainer" containerID="50ca62ec3fb587bd22466f206da0132db84f9ea73e44cd802dbd232beadf7577"
Oct 30 03:58:36.305496 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:36.305459    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bdwgx" event={"ID":"1fe5ff28-0896-4f54-b5cc-5326a0d60752","Type":"ContainerStarted","Data":"32b505541e843196f321566a1e875b15e1910d0fbd67d3d6ed385c79e9dac34b"}
Oct 30 03:58:41.751143 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:58:41.751092    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 03:58:45.348748 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:45.348666    5072 generic.go:356] "Generic (PLEG): container finished" podID="247ce32a-598a-4d86-9892-12af06a4c753" containerID="c54ea84650da135a603d64055f96b6713691f66bf810ad7c771849269b304d18" exitCode=0
Oct 30 03:58:45.348748 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:45.348740    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-crtnn-gt78p" event={"ID":"247ce32a-598a-4d86-9892-12af06a4c753","Type":"ContainerDied","Data":"c54ea84650da135a603d64055f96b6713691f66bf810ad7c771849269b304d18"}
Oct 30 03:58:46.517645 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.517620    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:58:46.539526 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.539496    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-trusted-ca-bundle\") pod \"247ce32a-598a-4d86-9892-12af06a4c753\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") "
Oct 30 03:58:46.539652 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.539581    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247ce32a-598a-4d86-9892-12af06a4c753-kube-api-access-ljf6f\" (UniqueName: \"kubernetes.io/projected/247ce32a-598a-4d86-9892-12af06a4c753-kube-api-access-ljf6f\") pod \"247ce32a-598a-4d86-9892-12af06a4c753\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") "
Oct 30 03:58:46.539652 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.539640    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-service-ca-bundle\") pod \"247ce32a-598a-4d86-9892-12af06a4c753\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") "
Oct 30 03:58:46.539726 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.539671    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247ce32a-598a-4d86-9892-12af06a4c753-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/247ce32a-598a-4d86-9892-12af06a4c753-archives-path\") pod \"247ce32a-598a-4d86-9892-12af06a4c753\" (UID: \"247ce32a-598a-4d86-9892-12af06a4c753\") "
Oct 30 03:58:46.540019 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.539990    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-trusted-ca-bundle" pod "247ce32a-598a-4d86-9892-12af06a4c753" (UID: "247ce32a-598a-4d86-9892-12af06a4c753"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:46.540019 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.540001    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-service-ca-bundle" pod "247ce32a-598a-4d86-9892-12af06a4c753" (UID: "247ce32a-598a-4d86-9892-12af06a4c753"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:46.540323 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.540044    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247ce32a-598a-4d86-9892-12af06a4c753-archives-path" pod "247ce32a-598a-4d86-9892-12af06a4c753" (UID: "247ce32a-598a-4d86-9892-12af06a4c753"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:46.541676 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.541654    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247ce32a-598a-4d86-9892-12af06a4c753-kube-api-access-ljf6f" pod "247ce32a-598a-4d86-9892-12af06a4c753" (UID: "247ce32a-598a-4d86-9892-12af06a4c753"). InnerVolumeSpecName "kube-api-access-ljf6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:46.641009 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.640930    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljf6f\" (UniqueName: \"kubernetes.io/projected/247ce32a-598a-4d86-9892-12af06a4c753-kube-api-access-ljf6f\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.641009 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.640959    5072 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-service-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.641009 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.640969    5072 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/247ce32a-598a-4d86-9892-12af06a4c753-archives-path\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.641009 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:46.640978    5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ce32a-598a-4d86-9892-12af06a4c753-trusted-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:47.358859 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:47.358827    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-crtnn-gt78p"
Oct 30 03:58:47.359028 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:47.358832    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-crtnn-gt78p" event={"ID":"247ce32a-598a-4d86-9892-12af06a4c753","Type":"ContainerDied","Data":"42a9081f3bd6a41c13de4d25ece9820f3aa80adc302aae6cfca37849fb556544"}
Oct 30 03:58:47.359028 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:47.358951    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42a9081f3bd6a41c13de4d25ece9820f3aa80adc302aae6cfca37849fb556544"
Oct 30 03:58:47.938862 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:47.938821    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-869d9c4d7-49r74" podUID="f84436e2-c830-4134-af93-5fc2f934ccbf" containerName="console" containerID="cri-o://bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26" gracePeriod=15
Oct 30 03:58:48.238959 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.238933    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:58:48.357330 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357296    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-console-config\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-console-config\") pod \"f84436e2-c830-4134-af93-5fc2f934ccbf\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") "
Oct 30 03:58:48.357330 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357334    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84436e2-c830-4134-af93-5fc2f934ccbf-kube-api-access-sntl5\" (UniqueName: \"kubernetes.io/projected/f84436e2-c830-4134-af93-5fc2f934ccbf-kube-api-access-sntl5\") pod \"f84436e2-c830-4134-af93-5fc2f934ccbf\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") "
Oct 30 03:58:48.357587 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357374    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-service-ca\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-service-ca\") pod \"f84436e2-c830-4134-af93-5fc2f934ccbf\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") "
Oct 30 03:58:48.357587 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357404    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-oauth-config\") pod \"f84436e2-c830-4134-af93-5fc2f934ccbf\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") "
Oct 30 03:58:48.357587 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357437    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-oauth-serving-cert\") pod \"f84436e2-c830-4134-af93-5fc2f934ccbf\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") "
Oct 30 03:58:48.357587 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357460    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-serving-cert\") pod \"f84436e2-c830-4134-af93-5fc2f934ccbf\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") "
Oct 30 03:58:48.357587 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357539    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-trusted-ca-bundle\") pod \"f84436e2-c830-4134-af93-5fc2f934ccbf\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") "
Oct 30 03:58:48.357587 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357585    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f84436e2-c830-4134-af93-5fc2f934ccbf-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84436e2-c830-4134-af93-5fc2f934ccbf-tmp\") pod \"f84436e2-c830-4134-af93-5fc2f934ccbf\" (UID: \"f84436e2-c830-4134-af93-5fc2f934ccbf\") "
Oct 30 03:58:48.357897 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357743    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-console-config" pod "f84436e2-c830-4134-af93-5fc2f934ccbf" (UID: "f84436e2-c830-4134-af93-5fc2f934ccbf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:48.357897 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357828    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-service-ca" pod "f84436e2-c830-4134-af93-5fc2f934ccbf" (UID: "f84436e2-c830-4134-af93-5fc2f934ccbf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:48.357996 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357963    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84436e2-c830-4134-af93-5fc2f934ccbf-tmp" pod "f84436e2-c830-4134-af93-5fc2f934ccbf" (UID: "f84436e2-c830-4134-af93-5fc2f934ccbf"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:48.357996 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357981    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-trusted-ca-bundle" pod "f84436e2-c830-4134-af93-5fc2f934ccbf" (UID: "f84436e2-c830-4134-af93-5fc2f934ccbf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:48.358170 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.357988    5072 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:48.358170 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.358020    5072 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-console-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:48.358170 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.358121    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-oauth-serving-cert" pod "f84436e2-c830-4134-af93-5fc2f934ccbf" (UID: "f84436e2-c830-4134-af93-5fc2f934ccbf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:48.359813 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.359790    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84436e2-c830-4134-af93-5fc2f934ccbf-kube-api-access-sntl5" pod "f84436e2-c830-4134-af93-5fc2f934ccbf" (UID: "f84436e2-c830-4134-af93-5fc2f934ccbf"). InnerVolumeSpecName "kube-api-access-sntl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:48.359941 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.359920    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-serving-cert" pod "f84436e2-c830-4134-af93-5fc2f934ccbf" (UID: "f84436e2-c830-4134-af93-5fc2f934ccbf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:48.360004 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.359965    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-oauth-config" pod "f84436e2-c830-4134-af93-5fc2f934ccbf" (UID: "f84436e2-c830-4134-af93-5fc2f934ccbf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:48.364908 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.364887    5072 generic.go:356] "Generic (PLEG): container finished" podID="f84436e2-c830-4134-af93-5fc2f934ccbf" containerID="bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26" exitCode=0
Oct 30 03:58:48.365007 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.364951    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869d9c4d7-49r74"
Oct 30 03:58:48.365007 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.364974    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869d9c4d7-49r74" event={"ID":"f84436e2-c830-4134-af93-5fc2f934ccbf","Type":"ContainerDied","Data":"bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26"}
Oct 30 03:58:48.365152 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.365043    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869d9c4d7-49r74" event={"ID":"f84436e2-c830-4134-af93-5fc2f934ccbf","Type":"ContainerDied","Data":"020d00fb0ee700473b02c072e0bf1339e5de17e52c2972317e21ce7812e05d8e"}
Oct 30 03:58:48.365152 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.365098    5072 scope.go:122] "RemoveContainer" containerID="bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26"
Oct 30 03:58:48.381953 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.381936    5072 scope.go:122] "RemoveContainer" containerID="bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26"
Oct 30 03:58:48.382332 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:58:48.382306    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26\": container with ID starting with bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26 not found: ID does not exist" containerID="bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26"
Oct 30 03:58:48.382408 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.382341    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26"} err="failed to get container status \"bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26\": rpc error: code = NotFound desc = could not find container \"bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26\": container with ID starting with bc3b05038f565dda1014f61328a7c8544a971688f7b49d8dfd4b387921fefa26 not found: ID does not exist"
Oct 30 03:58:48.417468 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.417449    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-869d9c4d7-49r74"]
Oct 30 03:58:48.421967 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.421947    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-869d9c4d7-49r74"]
Oct 30 03:58:48.458966 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.458941    5072 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-oauth-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:48.458966 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.458964    5072 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-oauth-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:48.459103 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.458974    5072 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84436e2-c830-4134-af93-5fc2f934ccbf-console-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:48.459103 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.458983    5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84436e2-c830-4134-af93-5fc2f934ccbf-trusted-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:48.459103 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.458992    5072 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f84436e2-c830-4134-af93-5fc2f934ccbf-tmp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:48.459103 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:48.459001    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sntl5\" (UniqueName: \"kubernetes.io/projected/f84436e2-c830-4134-af93-5fc2f934ccbf-kube-api-access-sntl5\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 03:58:50.410437 ip-10-0-139-99 kubenswrapper[5072]: I1030 03:58:50.410404    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84436e2-c830-4134-af93-5fc2f934ccbf" path="/var/lib/kubelet/pods/f84436e2-c830-4134-af93-5fc2f934ccbf/volumes"
Oct 30 03:59:11.750729 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:59:11.750650    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 03:59:41.748823 ip-10-0-139-99 kubenswrapper[5072]: E1030 03:59:41.748787    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:00:11.750179 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:00:11.750142    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:00:41.750208 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:00:41.750128    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:01:11.750149 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:01:11.750110    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:01:23.924638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.924602    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"]
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925102    5072 cpu_manager.go:414] "containerMap: removing container" podUID="247ce32a-598a-4d86-9892-12af06a4c753" containerName="insights-gathering"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925115    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247ce32a-598a-4d86-9892-12af06a4c753" containerName="insights-gathering"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925130    5072 cpu_manager.go:414] "containerMap: removing container" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="kube-rbac-proxy-metric"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925135    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="kube-rbac-proxy-metric"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925144    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="prometheus"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925149    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="prometheus"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925159    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="config-reloader"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925167    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="config-reloader"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925184    5072 cpu_manager.go:414] "containerMap: removing container" podUID="67dad9d6-9671-49f3-84a2-024463451062" containerName="insights-gathering"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925189    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67dad9d6-9671-49f3-84a2-024463451062" containerName="insights-gathering"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925200    5072 cpu_manager.go:414] "containerMap: removing container" podUID="63ac0921-404c-493d-9aec-02c93599b494" containerName="console"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925205    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ac0921-404c-493d-9aec-02c93599b494" containerName="console"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925213    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="init-config-reloader"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925217    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="init-config-reloader"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925224    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f84436e2-c830-4134-af93-5fc2f934ccbf" containerName="console"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925228    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84436e2-c830-4134-af93-5fc2f934ccbf" containerName="console"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925236    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="kube-rbac-proxy"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925240    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="kube-rbac-proxy"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925253    5072 cpu_manager.go:414] "containerMap: removing container" podUID="fe0bb808-beaa-4907-94e0-29b68fe8d121" containerName="console"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925260    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0bb808-beaa-4907-94e0-29b68fe8d121" containerName="console"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925267    5072 cpu_manager.go:414] "containerMap: removing container" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="kube-rbac-proxy-web"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925272    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="kube-rbac-proxy-web"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925277    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="kube-rbac-proxy-web"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925282    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="kube-rbac-proxy-web"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925292    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="thanos-sidecar"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925296    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="thanos-sidecar"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925310    5072 cpu_manager.go:414] "containerMap: removing container" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="config-reloader"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925315    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="config-reloader"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925327    5072 cpu_manager.go:414] "containerMap: removing container" podUID="343419d3-210d-4c12-86ff-c92f8f574b27" containerName="registry"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925335    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343419d3-210d-4c12-86ff-c92f8f574b27" containerName="registry"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925348    5072 cpu_manager.go:414] "containerMap: removing container" podUID="395d5cbd-c077-4578-b377-3034e42a2b94" containerName="registry"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925353    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="395d5cbd-c077-4578-b377-3034e42a2b94" containerName="registry"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925359    5072 cpu_manager.go:414] "containerMap: removing container" podUID="716a3fab-e121-4291-8ef5-02a47719ccbb" containerName="console"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925363    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="716a3fab-e121-4291-8ef5-02a47719ccbb" containerName="console"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925372    5072 cpu_manager.go:414] "containerMap: removing container" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="alertmanager"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925376    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="alertmanager"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925383    5072 cpu_manager.go:414] "containerMap: removing container" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="kube-rbac-proxy"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925387    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="kube-rbac-proxy"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925393    5072 cpu_manager.go:414] "containerMap: removing container" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="prom-label-proxy"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925413    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="prom-label-proxy"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925424    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="kube-rbac-proxy-thanos"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925429    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="kube-rbac-proxy-thanos"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925440    5072 cpu_manager.go:414] "containerMap: removing container" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="init-config-reloader"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925445    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="init-config-reloader"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925515    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="config-reloader"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925525    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="343419d3-210d-4c12-86ff-c92f8f574b27" containerName="registry"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925531    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="395d5cbd-c077-4578-b377-3034e42a2b94" containerName="registry"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925536    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="716a3fab-e121-4291-8ef5-02a47719ccbb" containerName="console"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925541    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="kube-rbac-proxy"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925546    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="alertmanager"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925551    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="prom-label-proxy"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925556    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="kube-rbac-proxy-thanos"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925561    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="config-reloader"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925568    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="247ce32a-598a-4d86-9892-12af06a4c753" containerName="insights-gathering"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925578    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="kube-rbac-proxy-metric"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925586    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="prometheus"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925594    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="67dad9d6-9671-49f3-84a2-024463451062" containerName="insights-gathering"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925599    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ac0921-404c-493d-9aec-02c93599b494" containerName="console"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925604    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="kube-rbac-proxy"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925608    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84436e2-c830-4134-af93-5fc2f934ccbf" containerName="console"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925613    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0bb808-beaa-4907-94e0-29b68fe8d121" containerName="console"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925618    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="497cda50-9fcd-4e3b-b7ab-93d9b599f9a7" containerName="kube-rbac-proxy-web"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925624    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="kube-rbac-proxy-web"
Oct 30 04:01:23.926949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.925629    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd8b824-8c21-40aa-9a1d-f3913690e556" containerName="thanos-sidecar"
Oct 30 04:01:23.929244 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.928855    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:23.960919 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.960892    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"]
Oct 30 04:01:23.996654 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.996629    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 04:01:23.996781 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.996713    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-8srh5\""
Oct 30 04:01:23.996781 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.996717    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 04:01:23.996781 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.996710    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 04:01:23.998689 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.998672    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 04:01:23.998765 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:23.998741    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 04:01:24.052948 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.052923    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2143dd12-cb22-46c5-a852-2d4268e1bc1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.053066 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.052950    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2143dd12-cb22-46c5-a852-2d4268e1bc1d-manifests\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.053066 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.052975    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2143dd12-cb22-46c5-a852-2d4268e1bc1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.053219 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.053197    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2143dd12-cb22-46c5-a852-2d4268e1bc1d-metrics-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.053277 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.053235    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcv6j\" (UniqueName: \"kubernetes.io/projected/2143dd12-cb22-46c5-a852-2d4268e1bc1d-kube-api-access-zcv6j\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.053373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.053358    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2143dd12-cb22-46c5-a852-2d4268e1bc1d-tmp\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.154476 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.154442    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2143dd12-cb22-46c5-a852-2d4268e1bc1d-metrics-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.154610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.154478    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcv6j\" (UniqueName: \"kubernetes.io/projected/2143dd12-cb22-46c5-a852-2d4268e1bc1d-kube-api-access-zcv6j\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.154610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.154522    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2143dd12-cb22-46c5-a852-2d4268e1bc1d-tmp\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.154610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.154560    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2143dd12-cb22-46c5-a852-2d4268e1bc1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.154610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.154587    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2143dd12-cb22-46c5-a852-2d4268e1bc1d-manifests\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.154890 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.154643    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2143dd12-cb22-46c5-a852-2d4268e1bc1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.155068 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.155035    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2143dd12-cb22-46c5-a852-2d4268e1bc1d-manifests\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.157220 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.157191    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2143dd12-cb22-46c5-a852-2d4268e1bc1d-tmp\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.157327 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.157218    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2143dd12-cb22-46c5-a852-2d4268e1bc1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.157327 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.157318    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2143dd12-cb22-46c5-a852-2d4268e1bc1d-metrics-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.157435 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.157345    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2143dd12-cb22-46c5-a852-2d4268e1bc1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.178333 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.178276    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcv6j\" (UniqueName: \"kubernetes.io/projected/2143dd12-cb22-46c5-a852-2d4268e1bc1d-kube-api-access-zcv6j\") pod \"opendatahub-operator-controller-manager-b54c6978-g9qqq\" (UID: \"2143dd12-cb22-46c5-a852-2d4268e1bc1d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.244721 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.244694    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:24.681193 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:01:24.681102    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2143dd12_cb22_46c5_a852_2d4268e1bc1d.slice/crio-aaddb6a5fed8425e95e683326b41fdb4177e3bcca49de1963ed12dc69bd81497 WatchSource:0}: Error finding container aaddb6a5fed8425e95e683326b41fdb4177e3bcca49de1963ed12dc69bd81497: Status 404 returned error can't find the container with id aaddb6a5fed8425e95e683326b41fdb4177e3bcca49de1963ed12dc69bd81497
Oct 30 04:01:24.684023 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:24.684001    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"]
Oct 30 04:01:25.066933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:25.066899    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq" event={"ID":"2143dd12-cb22-46c5-a852-2d4268e1bc1d","Type":"ContainerStarted","Data":"aaddb6a5fed8425e95e683326b41fdb4177e3bcca49de1963ed12dc69bd81497"}
Oct 30 04:01:28.086532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:28.086498    5072 generic.go:356] "Generic (PLEG): container finished" podID="2143dd12-cb22-46c5-a852-2d4268e1bc1d" containerID="293f600c0cf409a110d451731a95e24e4a36e14c8b080d860ff4bb3e42fd6413" exitCode=0
Oct 30 04:01:28.086951 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:28.086581    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq" event={"ID":"2143dd12-cb22-46c5-a852-2d4268e1bc1d","Type":"ContainerDied","Data":"293f600c0cf409a110d451731a95e24e4a36e14c8b080d860ff4bb3e42fd6413"}
Oct 30 04:01:29.096633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:29.096583    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq" event={"ID":"2143dd12-cb22-46c5-a852-2d4268e1bc1d","Type":"ContainerStarted","Data":"da9081c15b0cb55f599b69ebecee4d766389dde0707504c165a6ed115299d82a"}
Oct 30 04:01:29.097482 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:29.097462    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:29.210709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:29.210654    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq" podStartSLOduration=2.525950302 podStartE2EDuration="6.210639616s" totalImagesPullingTime="3.239398719s" totalInitContainerRuntime="445.290595ms" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:29.197285062 +0000 UTC m=+268.938708182" watchObservedRunningTime="2026-10-30 04:01:29.210639616 +0000 UTC m=+268.952062712"
Oct 30 04:01:34.108946 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.108914    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"]
Oct 30 04:01:34.115403 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.115379    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"
Oct 30 04:01:34.146639 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.146616    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-69b7b\""
Oct 30 04:01:34.146766 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.146616    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 04:01:34.146766 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.146628    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 04:01:34.149299 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.149278    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"]
Oct 30 04:01:34.253846 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.253820    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5j8\" (UniqueName: \"kubernetes.io/projected/0d158980-6b21-4602-9c06-846f1b6228bf-kube-api-access-nt5j8\") pod \"cert-manager-webhook-5dbbbf75cc-bd9vx\" (UID: \"0d158980-6b21-4602-9c06-846f1b6228bf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"
Oct 30 04:01:34.253973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.253959    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d158980-6b21-4602-9c06-846f1b6228bf-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bd9vx\" (UID: \"0d158980-6b21-4602-9c06-846f1b6228bf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"
Oct 30 04:01:34.355197 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.355170    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d158980-6b21-4602-9c06-846f1b6228bf-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bd9vx\" (UID: \"0d158980-6b21-4602-9c06-846f1b6228bf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"
Oct 30 04:01:34.355341 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.355218    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5j8\" (UniqueName: \"kubernetes.io/projected/0d158980-6b21-4602-9c06-846f1b6228bf-kube-api-access-nt5j8\") pod \"cert-manager-webhook-5dbbbf75cc-bd9vx\" (UID: \"0d158980-6b21-4602-9c06-846f1b6228bf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"
Oct 30 04:01:34.379614 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.379555    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d158980-6b21-4602-9c06-846f1b6228bf-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bd9vx\" (UID: \"0d158980-6b21-4602-9c06-846f1b6228bf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"
Oct 30 04:01:34.379714 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.379696    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5j8\" (UniqueName: \"kubernetes.io/projected/0d158980-6b21-4602-9c06-846f1b6228bf-kube-api-access-nt5j8\") pod \"cert-manager-webhook-5dbbbf75cc-bd9vx\" (UID: \"0d158980-6b21-4602-9c06-846f1b6228bf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"
Oct 30 04:01:34.441743 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.441714    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"
Oct 30 04:01:34.740004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:34.739911    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"]
Oct 30 04:01:35.125508 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:35.125427    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx" event={"ID":"0d158980-6b21-4602-9c06-846f1b6228bf","Type":"ContainerStarted","Data":"7b23de5f3b4ad6358c5cdf771787e035af9f0c9345430ae04d93bb809522596f"}
Oct 30 04:01:38.143565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:38.143522    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx" event={"ID":"0d158980-6b21-4602-9c06-846f1b6228bf","Type":"ContainerStarted","Data":"daa6309adb757a2b05ac3726f20cb2b13c713ed4054a5b422772d19fb4d6780a"}
Oct 30 04:01:38.143977 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:38.143699    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"
Oct 30 04:01:38.188584 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:38.188536    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx" podStartSLOduration=1.302845878 podStartE2EDuration="4.188523602s" totalImagesPullingTime="2.885677724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:38.188064875 +0000 UTC m=+277.929487972" watchObservedRunningTime="2026-10-30 04:01:38.188523602 +0000 UTC m=+277.929946677"
Oct 30 04:01:41.108395 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:41.108362    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-g9qqq"
Oct 30 04:01:41.750184 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:01:41.750151    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:01:44.149887 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:44.149852    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bd9vx"
Oct 30 04:01:46.756642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:46.756605    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5"]
Oct 30 04:01:46.762496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:46.762458    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5"
Oct 30 04:01:46.797272 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:46.797235    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tdgnf\""
Oct 30 04:01:46.806158 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:46.806115    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5"]
Oct 30 04:01:46.825363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:46.825330    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 04:01:46.825594 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:46.825408    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 04:01:46.875678 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:46.875642    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zckwx\" (UniqueName: \"kubernetes.io/projected/b85dd801-9b87-496b-b22a-ca557d3fd2c4-kube-api-access-zckwx\") pod \"limitador-operator-controller-manager-5778bb4449-mjlc5\" (UID: \"b85dd801-9b87-496b-b22a-ca557d3fd2c4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5"
Oct 30 04:01:46.976844 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:46.976806    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zckwx\" (UniqueName: \"kubernetes.io/projected/b85dd801-9b87-496b-b22a-ca557d3fd2c4-kube-api-access-zckwx\") pod \"limitador-operator-controller-manager-5778bb4449-mjlc5\" (UID: \"b85dd801-9b87-496b-b22a-ca557d3fd2c4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5"
Oct 30 04:01:47.006050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:47.006018    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zckwx\" (UniqueName: \"kubernetes.io/projected/b85dd801-9b87-496b-b22a-ca557d3fd2c4-kube-api-access-zckwx\") pod \"limitador-operator-controller-manager-5778bb4449-mjlc5\" (UID: \"b85dd801-9b87-496b-b22a-ca557d3fd2c4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5"
Oct 30 04:01:47.076548 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:47.076461    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5"
Oct 30 04:01:47.383048 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:47.382961    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5"]
Oct 30 04:01:47.387407 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:01:47.387379    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85dd801_9b87_496b_b22a_ca557d3fd2c4.slice/crio-758a495965ee0a9ead4265f061280f632be608bc8c97dac6a9ce2c9f127c078a WatchSource:0}: Error finding container 758a495965ee0a9ead4265f061280f632be608bc8c97dac6a9ce2c9f127c078a: Status 404 returned error can't find the container with id 758a495965ee0a9ead4265f061280f632be608bc8c97dac6a9ce2c9f127c078a
Oct 30 04:01:48.191918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:48.191883    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5" event={"ID":"b85dd801-9b87-496b-b22a-ca557d3fd2c4","Type":"ContainerStarted","Data":"758a495965ee0a9ead4265f061280f632be608bc8c97dac6a9ce2c9f127c078a"}
Oct 30 04:01:53.226298 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:53.226259    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5" event={"ID":"b85dd801-9b87-496b-b22a-ca557d3fd2c4","Type":"ContainerStarted","Data":"be638b535501c5c0005124c4ad8db8d59ff17762347eb4a8835a7a3f79970b50"}
Oct 30 04:01:53.226741 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:53.226345    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5"
Oct 30 04:01:53.270877 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:53.270819    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5" podStartSLOduration=1.9677039010000001 podStartE2EDuration="7.270806454s" totalImagesPullingTime="5.303102553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:53.269916214 +0000 UTC m=+293.011339312" watchObservedRunningTime="2026-10-30 04:01:53.270806454 +0000 UTC m=+293.012229550"
Oct 30 04:01:59.523719 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.523690    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"]
Oct 30 04:01:59.527730 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.527709    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"
Oct 30 04:01:59.547015 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.546984    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"]
Oct 30 04:01:59.559379 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.559349    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 04:01:59.559572 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.559373    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 04:01:59.559572 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.559419    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xs8bw\""
Oct 30 04:01:59.591209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.591175    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce064774-9d8f-4def-a363-809aae76d1ef-tmp\") pod \"kube-auth-proxy-685cdf9dcb-c2k7g\" (UID: \"ce064774-9d8f-4def-a363-809aae76d1ef\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"
Oct 30 04:01:59.591386 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.591228    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce064774-9d8f-4def-a363-809aae76d1ef-tls-certs\") pod \"kube-auth-proxy-685cdf9dcb-c2k7g\" (UID: \"ce064774-9d8f-4def-a363-809aae76d1ef\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"
Oct 30 04:01:59.591386 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.591364    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fp25\" (UniqueName: \"kubernetes.io/projected/ce064774-9d8f-4def-a363-809aae76d1ef-kube-api-access-6fp25\") pod \"kube-auth-proxy-685cdf9dcb-c2k7g\" (UID: \"ce064774-9d8f-4def-a363-809aae76d1ef\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"
Oct 30 04:01:59.692823 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.692788    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce064774-9d8f-4def-a363-809aae76d1ef-tmp\") pod \"kube-auth-proxy-685cdf9dcb-c2k7g\" (UID: \"ce064774-9d8f-4def-a363-809aae76d1ef\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"
Oct 30 04:01:59.693010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.692849    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce064774-9d8f-4def-a363-809aae76d1ef-tls-certs\") pod \"kube-auth-proxy-685cdf9dcb-c2k7g\" (UID: \"ce064774-9d8f-4def-a363-809aae76d1ef\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"
Oct 30 04:01:59.693010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.692897    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fp25\" (UniqueName: \"kubernetes.io/projected/ce064774-9d8f-4def-a363-809aae76d1ef-kube-api-access-6fp25\") pod \"kube-auth-proxy-685cdf9dcb-c2k7g\" (UID: \"ce064774-9d8f-4def-a363-809aae76d1ef\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"
Oct 30 04:01:59.695528 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.695497    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce064774-9d8f-4def-a363-809aae76d1ef-tmp\") pod \"kube-auth-proxy-685cdf9dcb-c2k7g\" (UID: \"ce064774-9d8f-4def-a363-809aae76d1ef\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"
Oct 30 04:01:59.695665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.695642    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce064774-9d8f-4def-a363-809aae76d1ef-tls-certs\") pod \"kube-auth-proxy-685cdf9dcb-c2k7g\" (UID: \"ce064774-9d8f-4def-a363-809aae76d1ef\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"
Oct 30 04:01:59.740696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.740667    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fp25\" (UniqueName: \"kubernetes.io/projected/ce064774-9d8f-4def-a363-809aae76d1ef-kube-api-access-6fp25\") pod \"kube-auth-proxy-685cdf9dcb-c2k7g\" (UID: \"ce064774-9d8f-4def-a363-809aae76d1ef\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"
Oct 30 04:01:59.842417 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:01:59.842331    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"
Oct 30 04:02:00.156021 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:00.155992    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g"]
Oct 30 04:02:00.157272 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:02:00.157245    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce064774_9d8f_4def_a363_809aae76d1ef.slice/crio-c9e909159e04408745124f45b5d4b98744ca24c87bbadaab081920afde8ecf14 WatchSource:0}: Error finding container c9e909159e04408745124f45b5d4b98744ca24c87bbadaab081920afde8ecf14: Status 404 returned error can't find the container with id c9e909159e04408745124f45b5d4b98744ca24c87bbadaab081920afde8ecf14
Oct 30 04:02:00.260491 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:00.260448    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g" event={"ID":"ce064774-9d8f-4def-a363-809aae76d1ef","Type":"ContainerStarted","Data":"c9e909159e04408745124f45b5d4b98744ca24c87bbadaab081920afde8ecf14"}
Oct 30 04:02:00.430235 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:00.430159    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:02:00.430381 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:00.430355    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:02:02.735291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:02.735260    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 04:02:03.287943 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:03.287890    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g" event={"ID":"ce064774-9d8f-4def-a363-809aae76d1ef","Type":"ContainerStarted","Data":"58f125ab7c6496d235da3c0fb2fc91b18a1fa59994c79f643d9521e847e0145d"}
Oct 30 04:02:03.332872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:03.332818    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-c2k7g" podStartSLOduration=1.772509109 podStartE2EDuration="4.332803989s" totalImagesPullingTime="2.56029488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:03.331993362 +0000 UTC m=+303.073416459" watchObservedRunningTime="2026-10-30 04:02:03.332803989 +0000 UTC m=+303.074227114"
Oct 30 04:02:04.018545 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.018512    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"]
Oct 30 04:02:04.022319 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.022300    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.042355 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.042331    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-wdrhj\""
Oct 30 04:02:04.057208 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.057182    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"]
Oct 30 04:02:04.068376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.068347    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 04:02:04.071189 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.071169    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 04:02:04.084066 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.084044    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 04:02:04.137322 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.137295    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3f48faf-a0f6-4a0a-ac98-963183990dab-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-qgl9w\" (UID: \"a3f48faf-a0f6-4a0a-ac98-963183990dab\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.137425 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.137328    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3f48faf-a0f6-4a0a-ac98-963183990dab-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-qgl9w\" (UID: \"a3f48faf-a0f6-4a0a-ac98-963183990dab\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.137425 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.137387    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f48faf-a0f6-4a0a-ac98-963183990dab-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-qgl9w\" (UID: \"a3f48faf-a0f6-4a0a-ac98-963183990dab\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.137532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.137517    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lg2h\" (UniqueName: \"kubernetes.io/projected/a3f48faf-a0f6-4a0a-ac98-963183990dab-kube-api-access-6lg2h\") pod \"kserve-module-controller-manager-7566cf5fc8-qgl9w\" (UID: \"a3f48faf-a0f6-4a0a-ac98-963183990dab\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.233210 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.233186    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mjlc5"
Oct 30 04:02:04.238872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.238851    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f48faf-a0f6-4a0a-ac98-963183990dab-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-qgl9w\" (UID: \"a3f48faf-a0f6-4a0a-ac98-963183990dab\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.238975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.238891    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lg2h\" (UniqueName: \"kubernetes.io/projected/a3f48faf-a0f6-4a0a-ac98-963183990dab-kube-api-access-6lg2h\") pod \"kserve-module-controller-manager-7566cf5fc8-qgl9w\" (UID: \"a3f48faf-a0f6-4a0a-ac98-963183990dab\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.239042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.238972    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3f48faf-a0f6-4a0a-ac98-963183990dab-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-qgl9w\" (UID: \"a3f48faf-a0f6-4a0a-ac98-963183990dab\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.239042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.239004    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3f48faf-a0f6-4a0a-ac98-963183990dab-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-qgl9w\" (UID: \"a3f48faf-a0f6-4a0a-ac98-963183990dab\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.239238 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.239216    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f48faf-a0f6-4a0a-ac98-963183990dab-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-qgl9w\" (UID: \"a3f48faf-a0f6-4a0a-ac98-963183990dab\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.239361 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.239340    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3f48faf-a0f6-4a0a-ac98-963183990dab-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-qgl9w\" (UID: \"a3f48faf-a0f6-4a0a-ac98-963183990dab\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.239885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.239862    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3f48faf-a0f6-4a0a-ac98-963183990dab-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-qgl9w\" (UID: \"a3f48faf-a0f6-4a0a-ac98-963183990dab\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.268302 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.268274    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lg2h\" (UniqueName: \"kubernetes.io/projected/a3f48faf-a0f6-4a0a-ac98-963183990dab-kube-api-access-6lg2h\") pod \"kserve-module-controller-manager-7566cf5fc8-qgl9w\" (UID: \"a3f48faf-a0f6-4a0a-ac98-963183990dab\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.335767 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.335673    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:04.664265 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:02:04.664237    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f48faf_a0f6_4a0a_ac98_963183990dab.slice/crio-7c5a64f711b89dc80c7d6f67711eb2671b0f5e6ce72ed862071bb905f926b3b7 WatchSource:0}: Error finding container 7c5a64f711b89dc80c7d6f67711eb2671b0f5e6ce72ed862071bb905f926b3b7: Status 404 returned error can't find the container with id 7c5a64f711b89dc80c7d6f67711eb2671b0f5e6ce72ed862071bb905f926b3b7
Oct 30 04:02:04.664426 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.664403    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"]
Oct 30 04:02:04.666441 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:04.666424    5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:02:05.302504 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:05.302462    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w" event={"ID":"a3f48faf-a0f6-4a0a-ac98-963183990dab","Type":"ContainerStarted","Data":"7c5a64f711b89dc80c7d6f67711eb2671b0f5e6ce72ed862071bb905f926b3b7"}
Oct 30 04:02:06.309135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:06.309103    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w" event={"ID":"a3f48faf-a0f6-4a0a-ac98-963183990dab","Type":"ContainerStarted","Data":"f944acd2e83a8bee6b762d0a5ea78f24278568072d8714d1a99bd366735cc0ce"}
Oct 30 04:02:06.309585 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:06.309245    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:06.367848 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:06.367736    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w" podStartSLOduration=1.961276928 podStartE2EDuration="3.367721814s" totalImagesPullingTime="1.406444886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:06.366168264 +0000 UTC m=+306.107591362" watchObservedRunningTime="2026-10-30 04:02:06.367721814 +0000 UTC m=+306.109144911"
Oct 30 04:02:11.751892 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:02:11.751813    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:02:27.315629 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:27.315598    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-qgl9w"
Oct 30 04:02:41.750970 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:02:41.750935    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:02:59.599913 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.599875    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b6fxf"]
Oct 30 04:02:59.605731 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.605710    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf"
Oct 30 04:02:59.621216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.621193    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b6fxf"]
Oct 30 04:02:59.642665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.642643    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"]
Oct 30 04:02:59.645253 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.645236    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"
Oct 30 04:02:59.666015 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.665989    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhjqx\" (UniqueName: \"kubernetes.io/projected/a80b6be8-ec4e-40c6-8889-256a744263b3-kube-api-access-hhjqx\") pod \"kserve-controller-manager-776cc66d47-b6fxf\" (UID: \"a80b6be8-ec4e-40c6-8889-256a744263b3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf"
Oct 30 04:02:59.666156 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.666136    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a80b6be8-ec4e-40c6-8889-256a744263b3-cert\") pod \"kserve-controller-manager-776cc66d47-b6fxf\" (UID: \"a80b6be8-ec4e-40c6-8889-256a744263b3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf"
Oct 30 04:02:59.712940 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.712900    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8rbbx\""
Oct 30 04:02:59.712940 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.712926    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 04:02:59.713187 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.712926    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-48tm7\""
Oct 30 04:02:59.713187 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.712972    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 04:02:59.727666 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.727617    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"]
Oct 30 04:02:59.767551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.767527    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d63b54f4-6e85-44a6-906d-80a0e2f6645c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5tr78\" (UID: \"d63b54f4-6e85-44a6-906d-80a0e2f6645c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"
Oct 30 04:02:59.767692 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.767595    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a80b6be8-ec4e-40c6-8889-256a744263b3-cert\") pod \"kserve-controller-manager-776cc66d47-b6fxf\" (UID: \"a80b6be8-ec4e-40c6-8889-256a744263b3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf"
Oct 30 04:02:59.767692 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.767673    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgmb\" (UniqueName: \"kubernetes.io/projected/d63b54f4-6e85-44a6-906d-80a0e2f6645c-kube-api-access-bwgmb\") pod \"llmisvc-controller-manager-869d5dcc5d-5tr78\" (UID: \"d63b54f4-6e85-44a6-906d-80a0e2f6645c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"
Oct 30 04:02:59.767762 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.767705    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhjqx\" (UniqueName: \"kubernetes.io/projected/a80b6be8-ec4e-40c6-8889-256a744263b3-kube-api-access-hhjqx\") pod \"kserve-controller-manager-776cc66d47-b6fxf\" (UID: \"a80b6be8-ec4e-40c6-8889-256a744263b3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf"
Oct 30 04:02:59.770274 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.770251    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a80b6be8-ec4e-40c6-8889-256a744263b3-cert\") pod \"kserve-controller-manager-776cc66d47-b6fxf\" (UID: \"a80b6be8-ec4e-40c6-8889-256a744263b3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf"
Oct 30 04:02:59.782874 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.782850    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-g9hm4"]
Oct 30 04:02:59.785447 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.785430    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4"
Oct 30 04:02:59.802849 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.802824    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhjqx\" (UniqueName: \"kubernetes.io/projected/a80b6be8-ec4e-40c6-8889-256a744263b3-kube-api-access-hhjqx\") pod \"kserve-controller-manager-776cc66d47-b6fxf\" (UID: \"a80b6be8-ec4e-40c6-8889-256a744263b3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf"
Oct 30 04:02:59.810842 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.810820    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rbw4r\""
Oct 30 04:02:59.810976 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.810873    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 04:02:59.812697 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.812678    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-g9hm4"]
Oct 30 04:02:59.869005 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.868936    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d63b54f4-6e85-44a6-906d-80a0e2f6645c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5tr78\" (UID: \"d63b54f4-6e85-44a6-906d-80a0e2f6645c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"
Oct 30 04:02:59.869173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.869068    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7hj5\" (UniqueName: \"kubernetes.io/projected/812448a2-1e32-4f19-86f0-8cf17f32edd1-kube-api-access-b7hj5\") pod \"model-serving-api-75bd89fdfc-g9hm4\" (UID: \"812448a2-1e32-4f19-86f0-8cf17f32edd1\") " pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4"
Oct 30 04:02:59.869246 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.869178    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgmb\" (UniqueName: \"kubernetes.io/projected/d63b54f4-6e85-44a6-906d-80a0e2f6645c-kube-api-access-bwgmb\") pod \"llmisvc-controller-manager-869d5dcc5d-5tr78\" (UID: \"d63b54f4-6e85-44a6-906d-80a0e2f6645c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"
Oct 30 04:02:59.869246 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.869216    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812448a2-1e32-4f19-86f0-8cf17f32edd1-tls-certs\") pod \"model-serving-api-75bd89fdfc-g9hm4\" (UID: \"812448a2-1e32-4f19-86f0-8cf17f32edd1\") " pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4"
Oct 30 04:02:59.871770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.871747    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d63b54f4-6e85-44a6-906d-80a0e2f6645c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5tr78\" (UID: \"d63b54f4-6e85-44a6-906d-80a0e2f6645c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"
Oct 30 04:02:59.893092 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.893051    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgmb\" (UniqueName: \"kubernetes.io/projected/d63b54f4-6e85-44a6-906d-80a0e2f6645c-kube-api-access-bwgmb\") pod \"llmisvc-controller-manager-869d5dcc5d-5tr78\" (UID: \"d63b54f4-6e85-44a6-906d-80a0e2f6645c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"
Oct 30 04:02:59.918146 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.918117    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf"
Oct 30 04:02:59.957591 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.957557    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"
Oct 30 04:02:59.970105 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.970026    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7hj5\" (UniqueName: \"kubernetes.io/projected/812448a2-1e32-4f19-86f0-8cf17f32edd1-kube-api-access-b7hj5\") pod \"model-serving-api-75bd89fdfc-g9hm4\" (UID: \"812448a2-1e32-4f19-86f0-8cf17f32edd1\") " pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4"
Oct 30 04:02:59.970212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.970115    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812448a2-1e32-4f19-86f0-8cf17f32edd1-tls-certs\") pod \"model-serving-api-75bd89fdfc-g9hm4\" (UID: \"812448a2-1e32-4f19-86f0-8cf17f32edd1\") " pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4"
Oct 30 04:02:59.973273 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.972924    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812448a2-1e32-4f19-86f0-8cf17f32edd1-tls-certs\") pod \"model-serving-api-75bd89fdfc-g9hm4\" (UID: \"812448a2-1e32-4f19-86f0-8cf17f32edd1\") " pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4"
Oct 30 04:02:59.994825 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:02:59.994788    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7hj5\" (UniqueName: \"kubernetes.io/projected/812448a2-1e32-4f19-86f0-8cf17f32edd1-kube-api-access-b7hj5\") pod \"model-serving-api-75bd89fdfc-g9hm4\" (UID: \"812448a2-1e32-4f19-86f0-8cf17f32edd1\") " pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4"
Oct 30 04:03:00.099586 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:00.099558    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4"
Oct 30 04:03:00.214897 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:00.214865    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b6fxf"]
Oct 30 04:03:00.216302 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:03:00.216277    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80b6be8_ec4e_40c6_8889_256a744263b3.slice/crio-58d97559b675b853743337da5d75110f327aba2e9f9b1da0a17f4ac1bb8b81cc WatchSource:0}: Error finding container 58d97559b675b853743337da5d75110f327aba2e9f9b1da0a17f4ac1bb8b81cc: Status 404 returned error can't find the container with id 58d97559b675b853743337da5d75110f327aba2e9f9b1da0a17f4ac1bb8b81cc
Oct 30 04:03:00.263762 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:00.263741    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"]
Oct 30 04:03:00.265412 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:03:00.265386    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd63b54f4_6e85_44a6_906d_80a0e2f6645c.slice/crio-d86bab5375913c32041835c7ba0c7dcdbba3534da00cc611e4a47dbb1835185a WatchSource:0}: Error finding container d86bab5375913c32041835c7ba0c7dcdbba3534da00cc611e4a47dbb1835185a: Status 404 returned error can't find the container with id d86bab5375913c32041835c7ba0c7dcdbba3534da00cc611e4a47dbb1835185a
Oct 30 04:03:00.402054 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:03:00.401974    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812448a2_1e32_4f19_86f0_8cf17f32edd1.slice/crio-b27f915133316d2ab91ab31b33c3cc434b9ade54153446740cae7b187d9493fc WatchSource:0}: Error finding container b27f915133316d2ab91ab31b33c3cc434b9ade54153446740cae7b187d9493fc: Status 404 returned error can't find the container with id b27f915133316d2ab91ab31b33c3cc434b9ade54153446740cae7b187d9493fc
Oct 30 04:03:00.408390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:00.408368    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-g9hm4"]
Oct 30 04:03:00.447712 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:00.447685    5072 scope.go:122] "RemoveContainer" containerID="e97abdeeb6ea2d5d5c7f770caf73c93d7fcd2711bf71f3f4d25ae5460d6b71f8"
Oct 30 04:03:00.586737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:00.586702    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78" event={"ID":"d63b54f4-6e85-44a6-906d-80a0e2f6645c","Type":"ContainerStarted","Data":"d86bab5375913c32041835c7ba0c7dcdbba3534da00cc611e4a47dbb1835185a"}
Oct 30 04:03:00.587872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:00.587847    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf" event={"ID":"a80b6be8-ec4e-40c6-8889-256a744263b3","Type":"ContainerStarted","Data":"58d97559b675b853743337da5d75110f327aba2e9f9b1da0a17f4ac1bb8b81cc"}
Oct 30 04:03:00.588973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:00.588951    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4" event={"ID":"812448a2-1e32-4f19-86f0-8cf17f32edd1","Type":"ContainerStarted","Data":"b27f915133316d2ab91ab31b33c3cc434b9ade54153446740cae7b187d9493fc"}
Oct 30 04:03:03.609209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:03.609170    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78" event={"ID":"d63b54f4-6e85-44a6-906d-80a0e2f6645c","Type":"ContainerStarted","Data":"1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16"}
Oct 30 04:03:03.609688 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:03.609258    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"
Oct 30 04:03:03.610955 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:03.610928    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf" event={"ID":"a80b6be8-ec4e-40c6-8889-256a744263b3","Type":"ContainerStarted","Data":"c1c6f1b3619ef6676cefbb17fd3cc5929151ccdd5dda048f44be994168d9a7b2"}
Oct 30 04:03:03.611103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:03.610984    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf"
Oct 30 04:03:03.612373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:03.612350    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4" event={"ID":"812448a2-1e32-4f19-86f0-8cf17f32edd1","Type":"ContainerStarted","Data":"0df4d24bf246528e235f2f06dffce876d38993cfba3a6ffead6e8ecc96bd76e7"}
Oct 30 04:03:03.612470 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:03.612404    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4"
Oct 30 04:03:03.652812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:03.652772    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78" podStartSLOduration=2.236766751 podStartE2EDuration="4.652759501s" totalImagesPullingTime="2.41599275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:03.651388887 +0000 UTC m=+363.392811986" watchObservedRunningTime="2026-10-30 04:03:03.652759501 +0000 UTC m=+363.394182598"
Oct 30 04:03:03.694518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:03.694461    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4" podStartSLOduration=2.510681205 podStartE2EDuration="4.694443232s" totalImagesPullingTime="2.183762027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:03.693658864 +0000 UTC m=+363.435081961" watchObservedRunningTime="2026-10-30 04:03:03.694443232 +0000 UTC m=+363.435866331"
Oct 30 04:03:03.737017 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:03.736971    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf" podStartSLOduration=2.123130112 podStartE2EDuration="4.736958412s" totalImagesPullingTime="2.6138283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:03.735610322 +0000 UTC m=+363.477033447" watchObservedRunningTime="2026-10-30 04:03:03.736958412 +0000 UTC m=+363.478381508"
Oct 30 04:03:11.750037 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:03:11.750003    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:03:14.620808 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:14.620770    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-g9hm4"
Oct 30 04:03:34.619931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:34.619885    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"
Oct 30 04:03:34.622876 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:03:34.622855    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-b6fxf"
Oct 30 04:03:41.751679 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:03:41.751599    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:04:11.750694 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:04:11.750659    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:04:41.750541 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:04:41.750505    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:05:11.750017 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:05:11.749925    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:05:41.750360 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:05:41.750319    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:06:11.750320 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:06:11.750286    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:06:41.750119 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:06:41.750013    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:07:00.464261 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:07:00.464235    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:07:00.466123 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:07:00.466103    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:07:11.750488 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:07:11.750445    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:07:41.750831 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:07:41.750794    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:08:11.749805 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:08:11.749725    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:08:41.751693 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:08:41.751655    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:09:11.749725 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:09:11.749690    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:09:41.751233 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:09:41.751152    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:10:11.751014 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:10:11.750979    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:10:41.749887 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:10:41.749849    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:11:11.751336 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:11:11.751244    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:11:41.750441 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:11:41.750337    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:12:00.498092 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:12:00.498052    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:12:00.501099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:12:00.501052    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:12:11.751346 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:12:11.751311    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:12:41.750824 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:12:41.750744    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:13:11.750559 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:13:11.750523    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:13:30.370097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.370053    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-598cbb96dc-ltrnh"]
Oct 30 04:13:30.373994 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.373974    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:30.391217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.391186    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 04:13:30.404336 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.404299    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j8pc2\""
Oct 30 04:13:30.404336 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.404302    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 04:13:30.404578 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.404564    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 04:13:30.408374 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.408345    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-598cbb96dc-ltrnh"]
Oct 30 04:13:30.470941 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.470908    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrxr\" (UniqueName: \"kubernetes.io/projected/20aee20e-32a1-4695-a39f-5bc9bdde31c1-kube-api-access-xjrxr\") pod \"maas-controller-598cbb96dc-ltrnh\" (UID: \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\") " pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:30.471111 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.470949    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-metrics-certs\") pod \"maas-controller-598cbb96dc-ltrnh\" (UID: \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\") " pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:30.471111 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.470981    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-webhook-cert\") pod \"maas-controller-598cbb96dc-ltrnh\" (UID: \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\") " pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:30.572281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.572243    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrxr\" (UniqueName: \"kubernetes.io/projected/20aee20e-32a1-4695-a39f-5bc9bdde31c1-kube-api-access-xjrxr\") pod \"maas-controller-598cbb96dc-ltrnh\" (UID: \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\") " pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:30.572281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.572286    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-metrics-certs\") pod \"maas-controller-598cbb96dc-ltrnh\" (UID: \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\") " pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:30.572497 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.572414    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-webhook-cert\") pod \"maas-controller-598cbb96dc-ltrnh\" (UID: \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\") " pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:30.574861 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.574843    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-metrics-certs\") pod \"maas-controller-598cbb96dc-ltrnh\" (UID: \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\") " pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:30.574924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.574879    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-webhook-cert\") pod \"maas-controller-598cbb96dc-ltrnh\" (UID: \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\") " pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:30.604613 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.604584    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrxr\" (UniqueName: \"kubernetes.io/projected/20aee20e-32a1-4695-a39f-5bc9bdde31c1-kube-api-access-xjrxr\") pod \"maas-controller-598cbb96dc-ltrnh\" (UID: \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\") " pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:30.695384 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.695362    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:30.909019 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.908983    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f4559f84c-tknlp"]
Oct 30 04:13:30.912985 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.912961    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:30.937956 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.937932    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f4559f84c-tknlp"]
Oct 30 04:13:30.976916 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.976852    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-metrics-certs\") pod \"maas-controller-6f4559f84c-tknlp\" (UID: \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\") " pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:30.977034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.976966    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgdkw\" (UniqueName: \"kubernetes.io/projected/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-kube-api-access-wgdkw\") pod \"maas-controller-6f4559f84c-tknlp\" (UID: \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\") " pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:30.977034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.976996    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-webhook-cert\") pod \"maas-controller-6f4559f84c-tknlp\" (UID: \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\") " pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:30.997987 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:30.997961    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-598cbb96dc-ltrnh"]
Oct 30 04:13:30.999031 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:13:30.999007    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20aee20e_32a1_4695_a39f_5bc9bdde31c1.slice/crio-d6a4302baad769f110a7890d7598cf8a023214daa5f09d7816fbeddd5d052f85 WatchSource:0}: Error finding container d6a4302baad769f110a7890d7598cf8a023214daa5f09d7816fbeddd5d052f85: Status 404 returned error can't find the container with id d6a4302baad769f110a7890d7598cf8a023214daa5f09d7816fbeddd5d052f85
Oct 30 04:13:31.000383 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:31.000362    5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:13:31.077500 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:31.077461    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgdkw\" (UniqueName: \"kubernetes.io/projected/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-kube-api-access-wgdkw\") pod \"maas-controller-6f4559f84c-tknlp\" (UID: \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\") " pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:31.077659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:31.077513    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-webhook-cert\") pod \"maas-controller-6f4559f84c-tknlp\" (UID: \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\") " pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:31.077659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:31.077577    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-metrics-certs\") pod \"maas-controller-6f4559f84c-tknlp\" (UID: \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\") " pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:31.080212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:31.080187    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-webhook-cert\") pod \"maas-controller-6f4559f84c-tknlp\" (UID: \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\") " pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:31.080212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:31.080211    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-metrics-certs\") pod \"maas-controller-6f4559f84c-tknlp\" (UID: \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\") " pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:31.101758 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:31.101732    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgdkw\" (UniqueName: \"kubernetes.io/projected/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-kube-api-access-wgdkw\") pod \"maas-controller-6f4559f84c-tknlp\" (UID: \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\") " pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:31.226656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:31.226624    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:31.528985 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:31.528899    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f4559f84c-tknlp"]
Oct 30 04:13:31.529821 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:13:31.529798    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ebfdc6_a13a_45ed_9ed8_e8cbaface1d7.slice/crio-46071c9cbdb1bdd93248a1453374fee2efb481c18c1b19f3b733d0fac4067d7b WatchSource:0}: Error finding container 46071c9cbdb1bdd93248a1453374fee2efb481c18c1b19f3b733d0fac4067d7b: Status 404 returned error can't find the container with id 46071c9cbdb1bdd93248a1453374fee2efb481c18c1b19f3b733d0fac4067d7b
Oct 30 04:13:31.692101 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:31.692005    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4559f84c-tknlp" event={"ID":"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7","Type":"ContainerStarted","Data":"46071c9cbdb1bdd93248a1453374fee2efb481c18c1b19f3b733d0fac4067d7b"}
Oct 30 04:13:31.693552 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:31.693514    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-598cbb96dc-ltrnh" event={"ID":"20aee20e-32a1-4695-a39f-5bc9bdde31c1","Type":"ContainerStarted","Data":"d6a4302baad769f110a7890d7598cf8a023214daa5f09d7816fbeddd5d052f85"}
Oct 30 04:13:34.718398 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:34.718364    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4559f84c-tknlp" event={"ID":"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7","Type":"ContainerStarted","Data":"cdc0e665c1047599ed6231045079daac0dd7daa02b4cc066aff27b1f9f4cdf51"}
Oct 30 04:13:34.718845 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:34.718500    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:34.720320 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:34.720288    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-598cbb96dc-ltrnh" event={"ID":"20aee20e-32a1-4695-a39f-5bc9bdde31c1","Type":"ContainerStarted","Data":"26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9"}
Oct 30 04:13:34.720445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:34.720347    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:34.768737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:34.768693    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6f4559f84c-tknlp" podStartSLOduration=2.098420727 podStartE2EDuration="4.768681142s" totalImagesPullingTime="2.670260415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:34.768406719 +0000 UTC m=+994.509829815" watchObservedRunningTime="2026-10-30 04:13:34.768681142 +0000 UTC m=+994.510104293"
Oct 30 04:13:34.818484 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:34.818436    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-598cbb96dc-ltrnh" podStartSLOduration=2.020484638 podStartE2EDuration="4.818423632s" totalImagesPullingTime="2.797938994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:34.816780953 +0000 UTC m=+994.558204051" watchObservedRunningTime="2026-10-30 04:13:34.818423632 +0000 UTC m=+994.559846730"
Oct 30 04:13:36.052227 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.052193    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"]
Oct 30 04:13:36.056288 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.056264    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.076193 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.076165    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 04:13:36.087740 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.087712    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 04:13:36.087865 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.087741    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-vxh9v\""
Oct 30 04:13:36.090175 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.089921    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"]
Oct 30 04:13:36.132204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.132179    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c7bec9d6-a8f4-4051-867a-b28db0efb1ed-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mb5bq\" (UID: \"c7bec9d6-a8f4-4051-867a-b28db0efb1ed\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.132340 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.132224    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bec9d6-a8f4-4051-867a-b28db0efb1ed-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mb5bq\" (UID: \"c7bec9d6-a8f4-4051-867a-b28db0efb1ed\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.132458 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.132379    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7bec9d6-a8f4-4051-867a-b28db0efb1ed-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mb5bq\" (UID: \"c7bec9d6-a8f4-4051-867a-b28db0efb1ed\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.132458 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.132410    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr4zt\" (UniqueName: \"kubernetes.io/projected/c7bec9d6-a8f4-4051-867a-b28db0efb1ed-kube-api-access-jr4zt\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mb5bq\" (UID: \"c7bec9d6-a8f4-4051-867a-b28db0efb1ed\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.176344 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.176321    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 04:13:36.232943 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.232918    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr4zt\" (UniqueName: \"kubernetes.io/projected/c7bec9d6-a8f4-4051-867a-b28db0efb1ed-kube-api-access-jr4zt\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mb5bq\" (UID: \"c7bec9d6-a8f4-4051-867a-b28db0efb1ed\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.233069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.232949    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c7bec9d6-a8f4-4051-867a-b28db0efb1ed-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mb5bq\" (UID: \"c7bec9d6-a8f4-4051-867a-b28db0efb1ed\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.233069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.232987    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bec9d6-a8f4-4051-867a-b28db0efb1ed-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mb5bq\" (UID: \"c7bec9d6-a8f4-4051-867a-b28db0efb1ed\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.233069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.233059    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7bec9d6-a8f4-4051-867a-b28db0efb1ed-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mb5bq\" (UID: \"c7bec9d6-a8f4-4051-867a-b28db0efb1ed\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.233631 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.233605    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7bec9d6-a8f4-4051-867a-b28db0efb1ed-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mb5bq\" (UID: \"c7bec9d6-a8f4-4051-867a-b28db0efb1ed\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.233850 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.233823    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c7bec9d6-a8f4-4051-867a-b28db0efb1ed-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mb5bq\" (UID: \"c7bec9d6-a8f4-4051-867a-b28db0efb1ed\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.235583 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.235567    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7bec9d6-a8f4-4051-867a-b28db0efb1ed-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mb5bq\" (UID: \"c7bec9d6-a8f4-4051-867a-b28db0efb1ed\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.256915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.256893    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr4zt\" (UniqueName: \"kubernetes.io/projected/c7bec9d6-a8f4-4051-867a-b28db0efb1ed-kube-api-access-jr4zt\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mb5bq\" (UID: \"c7bec9d6-a8f4-4051-867a-b28db0efb1ed\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.369748 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.369672    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:36.670177 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.670146    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"]
Oct 30 04:13:36.671276 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:13:36.671253    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bec9d6_a8f4_4051_867a_b28db0efb1ed.slice/crio-a7ac298dda200f36c913afdaffe1456eb79a710b50462cbcb5b64265ef50d375 WatchSource:0}: Error finding container a7ac298dda200f36c913afdaffe1456eb79a710b50462cbcb5b64265ef50d375: Status 404 returned error can't find the container with id a7ac298dda200f36c913afdaffe1456eb79a710b50462cbcb5b64265ef50d375
Oct 30 04:13:36.731870 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:36.731838    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq" event={"ID":"c7bec9d6-a8f4-4051-867a-b28db0efb1ed","Type":"ContainerStarted","Data":"a7ac298dda200f36c913afdaffe1456eb79a710b50462cbcb5b64265ef50d375"}
Oct 30 04:13:41.763049 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:13:41.763012    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:13:45.727671 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:45.727639    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:45.728151 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:45.727721    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 04:13:45.803698 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:45.803667    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-598cbb96dc-ltrnh"]
Oct 30 04:13:45.803990 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:45.803922    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-598cbb96dc-ltrnh" podUID="20aee20e-32a1-4695-a39f-5bc9bdde31c1" containerName="manager" containerID="cri-o://26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9" gracePeriod=30
Oct 30 04:13:46.088470 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.088443    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:46.238760 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.238670    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-metrics-certs\") pod \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\" (UID: \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\") "
Oct 30 04:13:46.238927 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.238778    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20aee20e-32a1-4695-a39f-5bc9bdde31c1-kube-api-access-xjrxr\" (UniqueName: \"kubernetes.io/projected/20aee20e-32a1-4695-a39f-5bc9bdde31c1-kube-api-access-xjrxr\") pod \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\" (UID: \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\") "
Oct 30 04:13:46.238927 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.238828    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-webhook-cert\" (UniqueName: \"kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-webhook-cert\") pod \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\" (UID: \"20aee20e-32a1-4695-a39f-5bc9bdde31c1\") "
Oct 30 04:13:46.241358 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.241322    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-metrics-certs" pod "20aee20e-32a1-4695-a39f-5bc9bdde31c1" (UID: "20aee20e-32a1-4695-a39f-5bc9bdde31c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:46.241576 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.241549    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-webhook-cert" pod "20aee20e-32a1-4695-a39f-5bc9bdde31c1" (UID: "20aee20e-32a1-4695-a39f-5bc9bdde31c1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:46.241638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.241570    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20aee20e-32a1-4695-a39f-5bc9bdde31c1-kube-api-access-xjrxr" pod "20aee20e-32a1-4695-a39f-5bc9bdde31c1" (UID: "20aee20e-32a1-4695-a39f-5bc9bdde31c1"). InnerVolumeSpecName "kube-api-access-xjrxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:13:46.340422 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.340381    5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-metrics-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:13:46.340422 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.340419    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjrxr\" (UniqueName: \"kubernetes.io/projected/20aee20e-32a1-4695-a39f-5bc9bdde31c1-kube-api-access-xjrxr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:13:46.340422 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.340431    5072 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20aee20e-32a1-4695-a39f-5bc9bdde31c1-webhook-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:13:46.797977 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.797941    5072 generic.go:356] "Generic (PLEG): container finished" podID="20aee20e-32a1-4695-a39f-5bc9bdde31c1" containerID="26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9" exitCode=0
Oct 30 04:13:46.798385 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.798005    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-598cbb96dc-ltrnh"
Oct 30 04:13:46.798385 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.798029    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-598cbb96dc-ltrnh" event={"ID":"20aee20e-32a1-4695-a39f-5bc9bdde31c1","Type":"ContainerDied","Data":"26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9"}
Oct 30 04:13:46.798385 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.798100    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-598cbb96dc-ltrnh" event={"ID":"20aee20e-32a1-4695-a39f-5bc9bdde31c1","Type":"ContainerDied","Data":"d6a4302baad769f110a7890d7598cf8a023214daa5f09d7816fbeddd5d052f85"}
Oct 30 04:13:46.798385 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.798123    5072 scope.go:122] "RemoveContainer" containerID="26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9"
Oct 30 04:13:46.807447 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.807252    5072 scope.go:122] "RemoveContainer" containerID="26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9"
Oct 30 04:13:46.807583 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:13:46.807565    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9\": container with ID starting with 26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9 not found: ID does not exist" containerID="26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9"
Oct 30 04:13:46.807627 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.807597    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9"} err="failed to get container status \"26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9\": rpc error: code = NotFound desc = could not find container \"26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9\": container with ID starting with 26fe79f47877cbe9c488a3058890afbb24dbcb5adb46ee5aa3c7d665a1e3ecc9 not found: ID does not exist"
Oct 30 04:13:46.850823 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.850797    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-598cbb96dc-ltrnh"]
Oct 30 04:13:46.864426 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:46.864400    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-598cbb96dc-ltrnh"]
Oct 30 04:13:47.804911 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:47.804874    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq" event={"ID":"c7bec9d6-a8f4-4051-867a-b28db0efb1ed","Type":"ContainerStarted","Data":"f06de7d8935a542e8089e38b2e746a2f730a7f018bf9a1c591e4faee5f934b3c"}
Oct 30 04:13:47.805412 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:47.804961    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:13:47.848318 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:47.848269    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq" podStartSLOduration=1.580748406 podStartE2EDuration="11.848254094s" totalImagesPullingTime="10.267505688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:47.846762083 +0000 UTC m=+1007.588185179" watchObservedRunningTime="2026-10-30 04:13:47.848254094 +0000 UTC m=+1007.589677230"
Oct 30 04:13:48.407387 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:48.407353    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20aee20e-32a1-4695-a39f-5bc9bdde31c1" path="/var/lib/kubelet/pods/20aee20e-32a1-4695-a39f-5bc9bdde31c1/volumes"
Oct 30 04:13:58.817023 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:13:58.816993    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mb5bq"
Oct 30 04:14:11.750795 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:14:11.750712    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:14:15.196516 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.196477    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"]
Oct 30 04:14:15.197001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.196985    5072 cpu_manager.go:414] "containerMap: removing container" podUID="20aee20e-32a1-4695-a39f-5bc9bdde31c1" containerName="manager"
Oct 30 04:14:15.197054 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.197001    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20aee20e-32a1-4695-a39f-5bc9bdde31c1" containerName="manager"
Oct 30 04:14:15.197122 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.197091    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="20aee20e-32a1-4695-a39f-5bc9bdde31c1" containerName="manager"
Oct 30 04:14:15.200357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.200333    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.234381 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.234337    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 04:14:15.234544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.234401    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 04:14:15.236347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.236321    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"]
Oct 30 04:14:15.299460 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.299436    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8cl5\" (UniqueName: \"kubernetes.io/projected/1e06d327-663e-402e-86be-c4e3296c37f3-kube-api-access-d8cl5\") pod \"maas-api-58d7f7c4f4-htctr\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.299593 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.299499    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-maas-api-tls\") pod \"maas-api-58d7f7c4f4-htctr\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.299593 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.299581    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e06d327-663e-402e-86be-c4e3296c37f3-openshift-service-ca\") pod \"maas-api-58d7f7c4f4-htctr\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.299683 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.299610    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-metrics-certs\") pod \"maas-api-58d7f7c4f4-htctr\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.306359 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.306341    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gw5bk\""
Oct 30 04:14:15.306495 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.306461    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 04:14:15.306495 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.306477    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 04:14:15.400574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.400548    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-maas-api-tls\") pod \"maas-api-58d7f7c4f4-htctr\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.400706 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.400583    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e06d327-663e-402e-86be-c4e3296c37f3-openshift-service-ca\") pod \"maas-api-58d7f7c4f4-htctr\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.400706 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.400601    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-metrics-certs\") pod \"maas-api-58d7f7c4f4-htctr\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.400706 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.400662    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8cl5\" (UniqueName: \"kubernetes.io/projected/1e06d327-663e-402e-86be-c4e3296c37f3-kube-api-access-d8cl5\") pod \"maas-api-58d7f7c4f4-htctr\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.401203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.401185    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e06d327-663e-402e-86be-c4e3296c37f3-openshift-service-ca\") pod \"maas-api-58d7f7c4f4-htctr\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.403170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.403151    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-maas-api-tls\") pod \"maas-api-58d7f7c4f4-htctr\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.403249 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.403203    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-metrics-certs\") pod \"maas-api-58d7f7c4f4-htctr\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.427664 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.427640    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8cl5\" (UniqueName: \"kubernetes.io/projected/1e06d327-663e-402e-86be-c4e3296c37f3-kube-api-access-d8cl5\") pod \"maas-api-58d7f7c4f4-htctr\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.513759 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.513681    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:15.813115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.813069    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"]
Oct 30 04:14:15.814272 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:14:15.814229    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e06d327_663e_402e_86be_c4e3296c37f3.slice/crio-e9d4ebbdcc74a07f2e63b06a0e8f27453cb6de7046ea3ab8d1df72c2d1aafa4a WatchSource:0}: Error finding container e9d4ebbdcc74a07f2e63b06a0e8f27453cb6de7046ea3ab8d1df72c2d1aafa4a: Status 404 returned error can't find the container with id e9d4ebbdcc74a07f2e63b06a0e8f27453cb6de7046ea3ab8d1df72c2d1aafa4a
Oct 30 04:14:15.949663 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:15.949633    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr" event={"ID":"1e06d327-663e-402e-86be-c4e3296c37f3","Type":"ContainerStarted","Data":"e9d4ebbdcc74a07f2e63b06a0e8f27453cb6de7046ea3ab8d1df72c2d1aafa4a"}
Oct 30 04:14:17.965227 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:17.965189    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr" event={"ID":"1e06d327-663e-402e-86be-c4e3296c37f3","Type":"ContainerStarted","Data":"3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf"}
Oct 30 04:14:17.965659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:17.965344    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:18.010036 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.009974    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr" podStartSLOduration=1.648054955 podStartE2EDuration="3.009960428s" totalImagesPullingTime="1.361905473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:18.008779433 +0000 UTC m=+1037.750202531" watchObservedRunningTime="2026-10-30 04:14:18.009960428 +0000 UTC m=+1037.751383524"
Oct 30 04:14:18.219399 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.219327    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"]
Oct 30 04:14:18.223615 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.223596    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"
Oct 30 04:14:18.242869 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.242848    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"]
Oct 30 04:14:18.329207 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.329177    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pckt\" (UniqueName: \"kubernetes.io/projected/6671eae0-dd1d-41f5-b479-2c8e36e14650-kube-api-access-5pckt\") pod \"llmisvc-controller-manager-6466c554d8-k6qmc\" (UID: \"6671eae0-dd1d-41f5-b479-2c8e36e14650\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"
Oct 30 04:14:18.329367 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.329251    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6671eae0-dd1d-41f5-b479-2c8e36e14650-cert\") pod \"llmisvc-controller-manager-6466c554d8-k6qmc\" (UID: \"6671eae0-dd1d-41f5-b479-2c8e36e14650\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"
Oct 30 04:14:18.430578 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.430544    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6671eae0-dd1d-41f5-b479-2c8e36e14650-cert\") pod \"llmisvc-controller-manager-6466c554d8-k6qmc\" (UID: \"6671eae0-dd1d-41f5-b479-2c8e36e14650\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"
Oct 30 04:14:18.430727 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.430616    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pckt\" (UniqueName: \"kubernetes.io/projected/6671eae0-dd1d-41f5-b479-2c8e36e14650-kube-api-access-5pckt\") pod \"llmisvc-controller-manager-6466c554d8-k6qmc\" (UID: \"6671eae0-dd1d-41f5-b479-2c8e36e14650\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"
Oct 30 04:14:18.433259 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.433238    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6671eae0-dd1d-41f5-b479-2c8e36e14650-cert\") pod \"llmisvc-controller-manager-6466c554d8-k6qmc\" (UID: \"6671eae0-dd1d-41f5-b479-2c8e36e14650\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"
Oct 30 04:14:18.456991 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.456964    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pckt\" (UniqueName: \"kubernetes.io/projected/6671eae0-dd1d-41f5-b479-2c8e36e14650-kube-api-access-5pckt\") pod \"llmisvc-controller-manager-6466c554d8-k6qmc\" (UID: \"6671eae0-dd1d-41f5-b479-2c8e36e14650\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"
Oct 30 04:14:18.536824 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.536766    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"
Oct 30 04:14:18.837140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.837118    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"]
Oct 30 04:14:18.838345 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:14:18.838318    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6671eae0_dd1d_41f5_b479_2c8e36e14650.slice/crio-e20e9ae6f6f301e735759277893040a0ef2a686001f3d336568dc3ee9732f3b9 WatchSource:0}: Error finding container e20e9ae6f6f301e735759277893040a0ef2a686001f3d336568dc3ee9732f3b9: Status 404 returned error can't find the container with id e20e9ae6f6f301e735759277893040a0ef2a686001f3d336568dc3ee9732f3b9
Oct 30 04:14:18.971773 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:18.971740    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc" event={"ID":"6671eae0-dd1d-41f5-b479-2c8e36e14650","Type":"ContainerStarted","Data":"e20e9ae6f6f301e735759277893040a0ef2a686001f3d336568dc3ee9732f3b9"}
Oct 30 04:14:19.978451 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:19.978408    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc" event={"ID":"6671eae0-dd1d-41f5-b479-2c8e36e14650","Type":"ContainerStarted","Data":"da3759740d88104d13e9d54592530ba9520716d7cd9e41dd5cdf4c39efb308ae"}
Oct 30 04:14:19.978923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:19.978540    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"
Oct 30 04:14:20.020826 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:20.020760    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc" podStartSLOduration=1.695255816 podStartE2EDuration="2.020745715s" totalImagesPullingTime="325.489899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:20.020020274 +0000 UTC m=+1039.761443372" watchObservedRunningTime="2026-10-30 04:14:20.020745715 +0000 UTC m=+1039.762168873"
Oct 30 04:14:23.977635 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:23.977605    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 04:14:33.495610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.495582    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"]
Oct 30 04:14:33.499789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.499769    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.528314 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.528288    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 04:14:33.530350 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.530329    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"]
Oct 30 04:14:33.588652 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.588625    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.588787 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.588673    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.588787 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.588752    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f813b35-b180-40dd-bdd1-91fffa7e133e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.588908 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.588889    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.588951 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.588929    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.589003 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.588956    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzd4z\" (UniqueName: \"kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-kube-api-access-lzd4z\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.589103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.589056    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.589167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.589120    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.589211 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.589182    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.602962 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.602941    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vwdzb\""
Oct 30 04:14:33.690161 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690138    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.690301 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690170    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.690301 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690193    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.690301 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690211    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.690301 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690247    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.690301 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690270    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f813b35-b180-40dd-bdd1-91fffa7e133e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.690551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690355    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.690551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690401    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.690551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690438    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzd4z\" (UniqueName: \"kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-kube-api-access-lzd4z\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.690707 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690607    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.690707 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690687    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.690820 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690797    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.690877 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.690841    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.691034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.691015    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f813b35-b180-40dd-bdd1-91fffa7e133e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.692827 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.692805    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.693180 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.693149    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.721795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.721776    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzd4z\" (UniqueName: \"kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-kube-api-access-lzd4z\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.721985 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.721969    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-frvgg\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:33.814709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:33.814660    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:34.129465 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:34.129435    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"]
Oct 30 04:14:34.130117 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:14:34.130068    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f813b35_b180_40dd_bdd1_91fffa7e133e.slice/crio-96ef6bebdd042b5bfa2c8a9528b84cb7be70ed8e60dfda32a5b4958e1f6eb322 WatchSource:0}: Error finding container 96ef6bebdd042b5bfa2c8a9528b84cb7be70ed8e60dfda32a5b4958e1f6eb322: Status 404 returned error can't find the container with id 96ef6bebdd042b5bfa2c8a9528b84cb7be70ed8e60dfda32a5b4958e1f6eb322
Oct 30 04:14:35.057553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:35.057517    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg" event={"ID":"9f813b35-b180-40dd-bdd1-91fffa7e133e","Type":"ContainerStarted","Data":"96ef6bebdd042b5bfa2c8a9528b84cb7be70ed8e60dfda32a5b4958e1f6eb322"}
Oct 30 04:14:36.801581 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:36.801550    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:14:36.801882 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:36.801616    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:14:36.801882 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:36.801651    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:14:37.070906 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:37.070815    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg" event={"ID":"9f813b35-b180-40dd-bdd1-91fffa7e133e","Type":"ContainerStarted","Data":"d810ba66e358f5b7a5e1259a2b5e9bfa89349bb45b8b97bb8bd2ea4765d181da"}
Oct 30 04:14:37.143005 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:37.142953    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg" podStartSLOduration=1.474403133 podStartE2EDuration="4.142938992s" totalImagesPullingTime="2.668535859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:37.12955065 +0000 UTC m=+1056.870973748" watchObservedRunningTime="2026-10-30 04:14:37.142938992 +0000 UTC m=+1056.884362088"
Oct 30 04:14:37.815610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:37.815569    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:37.817159 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:37.817131    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-frvgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 04:14:37.817280 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:37.817186    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg" podUID="9f813b35-b180-40dd-bdd1-91fffa7e133e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 04:14:38.815086 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:38.815040    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-frvgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 04:14:38.815245 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:38.815105    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg" podUID="9f813b35-b180-40dd-bdd1-91fffa7e133e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 04:14:39.815143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:39.815109    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-frvgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 04:14:39.815541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:39.815166    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg" podUID="9f813b35-b180-40dd-bdd1-91fffa7e133e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 04:14:40.815867 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:40.815835    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-frvgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 04:14:40.816302 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:40.815886    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg" podUID="9f813b35-b180-40dd-bdd1-91fffa7e133e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 04:14:41.751613 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:14:41.751579    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:14:41.815900 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:41.815867    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-frvgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 04:14:41.816310 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:41.815920    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg" podUID="9f813b35-b180-40dd-bdd1-91fffa7e133e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 04:14:42.819176 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:42.819147    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:42.819557 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:42.819505    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:42.820140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:42.820120    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:14:48.543271 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.543236    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"]
Oct 30 04:14:48.547741 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.547718    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.572531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.572496    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"]
Oct 30 04:14:48.654370 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.654342    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.654511 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.654392    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.654511 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.654412    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.654591 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.654513    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.654591 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.654551    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.654591 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.654579    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.654695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.654602    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.654741 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.654722    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.654775 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.654751    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfcbc\" (UniqueName: \"kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-kube-api-access-tfcbc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.755525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.755480    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.755695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.755535    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfcbc\" (UniqueName: \"kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-kube-api-access-tfcbc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.755695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.755656    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.755820 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.755726    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.755820 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.755756    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.755820 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.755806    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.755969 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.755854    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.755969 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.755888    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.755969 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.755923    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.756264 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.756244    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.756344 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.756283    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.756411 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.756368    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.756411 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.756384    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.756730 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.756710    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.758795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.758773    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.759185 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.759165    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.779710 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.779682    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.780041 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.780019    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfcbc\" (UniqueName: \"kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-kube-api-access-tfcbc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flqhx\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:48.861714 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:48.861658    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:49.182899 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:49.182870    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"]
Oct 30 04:14:49.183548 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:14:49.183520    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f88e7d_21c3_4e59_9665_5cb24a87b5d5.slice/crio-9e64b02ee496479dc5659620e79aaf39f655eff35a2f20a9af31964c1a55d37c WatchSource:0}: Error finding container 9e64b02ee496479dc5659620e79aaf39f655eff35a2f20a9af31964c1a55d37c: Status 404 returned error can't find the container with id 9e64b02ee496479dc5659620e79aaf39f655eff35a2f20a9af31964c1a55d37c
Oct 30 04:14:49.186716 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:49.186681    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:14:49.186826 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:49.186761    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:14:49.186826 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:49.186806    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:14:50.148794 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:50.148762    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx" event={"ID":"27f88e7d-21c3-4e59-9665-5cb24a87b5d5","Type":"ContainerStarted","Data":"ff6d95cfc32516967d9e1396ced5a116b3a0a7f9601d0268e0fa72b9b77aaa4a"}
Oct 30 04:14:50.149216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:50.148811    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx" event={"ID":"27f88e7d-21c3-4e59-9665-5cb24a87b5d5","Type":"ContainerStarted","Data":"9e64b02ee496479dc5659620e79aaf39f655eff35a2f20a9af31964c1a55d37c"}
Oct 30 04:14:50.212123 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:50.212042    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx" podStartSLOduration=2.212029704 podStartE2EDuration="2.212029704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:50.209978316 +0000 UTC m=+1069.951401416" watchObservedRunningTime="2026-10-30 04:14:50.212029704 +0000 UTC m=+1069.953452800"
Oct 30 04:14:50.874103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:50.873156    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:50.877868 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:50.877841    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-flqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:14:50.878116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:50.878057    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx" podUID="27f88e7d-21c3-4e59-9665-5cb24a87b5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 04:14:50.986029 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:50.986000    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k6qmc"
Oct 30 04:14:51.091707 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:51.091663    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"]
Oct 30 04:14:51.091969 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:51.091941    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78" podUID="d63b54f4-6e85-44a6-906d-80a0e2f6645c" containerName="manager" containerID="cri-o://1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16" gracePeriod=30
Oct 30 04:14:51.388407 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:51.388376    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"
Oct 30 04:14:51.499862 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:51.499777    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63b54f4-6e85-44a6-906d-80a0e2f6645c-kube-api-access-bwgmb\" (UniqueName: \"kubernetes.io/projected/d63b54f4-6e85-44a6-906d-80a0e2f6645c-kube-api-access-bwgmb\") pod \"d63b54f4-6e85-44a6-906d-80a0e2f6645c\" (UID: \"d63b54f4-6e85-44a6-906d-80a0e2f6645c\") "
Oct 30 04:14:51.499862 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:51.499855    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d63b54f4-6e85-44a6-906d-80a0e2f6645c-cert\" (UniqueName: \"kubernetes.io/secret/d63b54f4-6e85-44a6-906d-80a0e2f6645c-cert\") pod \"d63b54f4-6e85-44a6-906d-80a0e2f6645c\" (UID: \"d63b54f4-6e85-44a6-906d-80a0e2f6645c\") "
Oct 30 04:14:51.502062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:51.502031    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63b54f4-6e85-44a6-906d-80a0e2f6645c-kube-api-access-bwgmb" pod "d63b54f4-6e85-44a6-906d-80a0e2f6645c" (UID: "d63b54f4-6e85-44a6-906d-80a0e2f6645c"). InnerVolumeSpecName "kube-api-access-bwgmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:51.502062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:51.502038    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d63b54f4-6e85-44a6-906d-80a0e2f6645c-cert" pod "d63b54f4-6e85-44a6-906d-80a0e2f6645c" (UID: "d63b54f4-6e85-44a6-906d-80a0e2f6645c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:51.600774 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:51.600739    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwgmb\" (UniqueName: \"kubernetes.io/projected/d63b54f4-6e85-44a6-906d-80a0e2f6645c-kube-api-access-bwgmb\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:14:51.600774 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:51.600769    5072 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d63b54f4-6e85-44a6-906d-80a0e2f6645c-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:14:51.865979 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:51.865470    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-flqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:14:51.865979 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:51.865528    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx" podUID="27f88e7d-21c3-4e59-9665-5cb24a87b5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 04:14:52.188188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:52.188153    5072 generic.go:356] "Generic (PLEG): container finished" podID="d63b54f4-6e85-44a6-906d-80a0e2f6645c" containerID="1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16" exitCode=0
Oct 30 04:14:52.188386 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:52.188252    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"
Oct 30 04:14:52.188386 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:52.188247    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78" event={"ID":"d63b54f4-6e85-44a6-906d-80a0e2f6645c","Type":"ContainerDied","Data":"1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16"}
Oct 30 04:14:52.188505 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:52.188388    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78" event={"ID":"d63b54f4-6e85-44a6-906d-80a0e2f6645c","Type":"ContainerDied","Data":"d86bab5375913c32041835c7ba0c7dcdbba3534da00cc611e4a47dbb1835185a"}
Oct 30 04:14:52.188505 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:52.188420    5072 scope.go:122] "RemoveContainer" containerID="1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16"
Oct 30 04:14:52.202347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:52.202322    5072 scope.go:122] "RemoveContainer" containerID="1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16"
Oct 30 04:14:52.202854 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:14:52.202821    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16\": container with ID starting with 1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16 not found: ID does not exist" containerID="1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16"
Oct 30 04:14:52.202970 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:52.202867    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16"} err="failed to get container status \"1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16\": rpc error: code = NotFound desc = could not find container \"1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16\": container with ID starting with 1db8c510a328e41374c752226055976bb33030f474843ea150e7c9b741720c16 not found: ID does not exist"
Oct 30 04:14:52.243632 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:52.243605    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"]
Oct 30 04:14:52.245557 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:52.245536    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5tr78"]
Oct 30 04:14:52.406902 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:52.406867    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63b54f4-6e85-44a6-906d-80a0e2f6645c" path="/var/lib/kubelet/pods/d63b54f4-6e85-44a6-906d-80a0e2f6645c/volumes"
Oct 30 04:14:52.870893 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:52.870438    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-flqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:14:52.870893 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:52.870511    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx" podUID="27f88e7d-21c3-4e59-9665-5cb24a87b5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 04:14:53.871099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:53.863413    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-flqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:14:53.871099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:53.863471    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx" podUID="27f88e7d-21c3-4e59-9665-5cb24a87b5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 04:14:54.876103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:54.875528    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-flqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:14:54.876103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:54.875596    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx" podUID="27f88e7d-21c3-4e59-9665-5cb24a87b5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 04:14:55.866515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:55.866489    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:55.866786 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:55.866764    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:14:55.867372 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:14:55.867356    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:15:03.627803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.627769    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"]
Oct 30 04:15:03.628565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.628544    5072 cpu_manager.go:414] "containerMap: removing container" podUID="d63b54f4-6e85-44a6-906d-80a0e2f6645c" containerName="manager"
Oct 30 04:15:03.628615 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.628568    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63b54f4-6e85-44a6-906d-80a0e2f6645c" containerName="manager"
Oct 30 04:15:03.628778 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.628764    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63b54f4-6e85-44a6-906d-80a0e2f6645c" containerName="manager"
Oct 30 04:15:03.634202 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.634180    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.669171 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.669139    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"]
Oct 30 04:15:03.715935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.715902    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.716055 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.715950    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.716144 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.716098    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.716144 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.716135    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.716223 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.716159    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.716259 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.716246    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.716302 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.716288    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trnkc\" (UniqueName: \"kubernetes.io/projected/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-kube-api-access-trnkc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.716336 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.716322    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.716373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.716342    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.816836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.816800    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.816980 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.816858    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnkc\" (UniqueName: \"kubernetes.io/projected/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-kube-api-access-trnkc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.816980 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.816886    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.816980 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.816909    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.816980 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.816961    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.817237 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.817004    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.817237 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.817060    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.817237 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.817114    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.817237 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.817147    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.817440 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.817250    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.817440 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.817397    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.817531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.817468    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.817531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.817492    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.818059 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.817978    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.819362 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.819339    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.819672 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.819657    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.841703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.841678    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trnkc\" (UniqueName: \"kubernetes.io/projected/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-kube-api-access-trnkc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.841805 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.841716    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b748c4e2-2e95-43e6-8dc4-9bf3958a24d2-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-txlk4\" (UID: \"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:03.948140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:03.948120    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:04.265023 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:15:04.264997    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb748c4e2_2e95_43e6_8dc4_9bf3958a24d2.slice/crio-841a99819ea634a74401dd794eff072c573630877c1348a9ab2573f30570d62a WatchSource:0}: Error finding container 841a99819ea634a74401dd794eff072c573630877c1348a9ab2573f30570d62a: Status 404 returned error can't find the container with id 841a99819ea634a74401dd794eff072c573630877c1348a9ab2573f30570d62a
Oct 30 04:15:04.265258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:04.265237    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"]
Oct 30 04:15:04.267226 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:04.267196    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:15:04.267308 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:04.267283    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:15:04.267356 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:04.267313    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:15:05.285247 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:05.285203    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4" event={"ID":"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2","Type":"ContainerStarted","Data":"ab22598c607424b773143a8cc59d07ef04019b27e1460c3443118cdddda6ee3a"}
Oct 30 04:15:05.285649 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:05.285271    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4" event={"ID":"b748c4e2-2e95-43e6-8dc4-9bf3958a24d2","Type":"ContainerStarted","Data":"841a99819ea634a74401dd794eff072c573630877c1348a9ab2573f30570d62a"}
Oct 30 04:15:05.347135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:05.347057    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4" podStartSLOduration=2.347043725 podStartE2EDuration="2.347043725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:05.345429203 +0000 UTC m=+1085.086852301" watchObservedRunningTime="2026-10-30 04:15:05.347043725 +0000 UTC m=+1085.088466821"
Oct 30 04:15:05.948339 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:05.948307    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:05.949982 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:05.949952    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-txlk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:15:05.950138 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:05.950002    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4" podUID="b748c4e2-2e95-43e6-8dc4-9bf3958a24d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 04:15:06.948610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:06.948578    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-txlk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:15:06.949004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:06.948630    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4" podUID="b748c4e2-2e95-43e6-8dc4-9bf3958a24d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 04:15:07.949187 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:07.949153    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-txlk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:15:07.949543 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:07.949202    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4" podUID="b748c4e2-2e95-43e6-8dc4-9bf3958a24d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 04:15:08.948550 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:08.948515    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-txlk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:15:08.948725 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:08.948573    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4" podUID="b748c4e2-2e95-43e6-8dc4-9bf3958a24d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 04:15:09.949177 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:09.949145    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-txlk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:15:09.949560 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:09.949194    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4" podUID="b748c4e2-2e95-43e6-8dc4-9bf3958a24d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 04:15:10.952730 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:10.952696    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:10.953164 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:10.953144    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:10.953721 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:15:10.953707    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4"
Oct 30 04:15:11.751698 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:15:11.751664    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:15:41.751855 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:15:41.751763    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:16:11.750888 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:16:11.750850    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:16:41.751543 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:16:41.751426    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:17:00.538441 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:17:00.538415    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:17:00.542174 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:17:00.542144    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:17:11.751416 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:17:11.751323    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:17:41.753219 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:17:41.753181    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:18:01.743714 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.743682    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"]
Oct 30 04:18:01.748160 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.748137    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.776643 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.776610    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-36bbac\"/\"default-dockercfg-llsp2\""
Oct 30 04:18:01.776824 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.776656    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-36bbac\"/\"openshift-service-ca.crt\""
Oct 30 04:18:01.776824 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.776653    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-36bbac\"/\"kube-root-ca.crt\""
Oct 30 04:18:01.777287 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.777266    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-36bbac\"/\"simulator-w0-36bbac-kserve-self-signed-certs\""
Oct 30 04:18:01.778952 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.778932    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"]
Oct 30 04:18:01.839396 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.839363    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-model-cache\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.839396 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.839401    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tls-certs\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.839652 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.839505    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-dshm\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.839652 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.839563    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-home\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.839762 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.839659    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtcqb\" (UniqueName: \"kubernetes.io/projected/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-kube-api-access-qtcqb\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.839824 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.839768    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tmp-dir\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.941328 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.941289    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-home\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.941515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.941351    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtcqb\" (UniqueName: \"kubernetes.io/projected/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-kube-api-access-qtcqb\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.941515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.941423    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tmp-dir\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.941515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.941512    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-model-cache\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.941739 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.941650    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tls-certs\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.941801 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.941763    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-dshm\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.941861 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.941802    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-home\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.941861 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.941826    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-model-cache\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.941998 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.941976    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tmp-dir\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.944290 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.944267    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-dshm\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.944584 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.944560    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tls-certs\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:01.966314 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:01.966279    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtcqb\" (UniqueName: \"kubernetes.io/projected/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-kube-api-access-qtcqb\") pod \"simulator-w0-36bbac-kserve-7584f46d-stfr8\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") " pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:02.061537 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:02.061453    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:02.367861 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:02.367831    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"]
Oct 30 04:18:02.369000 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:18:02.368975    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc6cf57f_0ddb_4e76_a06b_1cddd5e24070.slice/crio-93903597b64e5d9dadb7c0144456ecbd9888ae1dddb21b3fc4a1301756402d23 WatchSource:0}: Error finding container 93903597b64e5d9dadb7c0144456ecbd9888ae1dddb21b3fc4a1301756402d23: Status 404 returned error can't find the container with id 93903597b64e5d9dadb7c0144456ecbd9888ae1dddb21b3fc4a1301756402d23
Oct 30 04:18:03.331513 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:03.331477    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8" event={"ID":"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070","Type":"ContainerStarted","Data":"93903597b64e5d9dadb7c0144456ecbd9888ae1dddb21b3fc4a1301756402d23"}
Oct 30 04:18:04.339113 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:04.339048    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8" event={"ID":"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070","Type":"ContainerStarted","Data":"482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591"}
Oct 30 04:18:04.383519 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:04.383471    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8" podStartSLOduration=2.353919337 podStartE2EDuration="3.383454829s" totalImagesPullingTime="1.029535492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:04.381558672 +0000 UTC m=+1264.122981770" watchObservedRunningTime="2026-10-30 04:18:04.383454829 +0000 UTC m=+1264.124877927"
Oct 30 04:18:05.062112 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:05.062026    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:11.751205 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:18:11.751153    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:18:12.062649 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:12.062571    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:12.075598 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:12.075562    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:12.398358 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:12.398289    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 04:18:21.483151 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.483116    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"]
Oct 30 04:18:21.487794 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.487773    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.513503 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.513428    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"openshift-service-ca.crt\""
Oct 30 04:18:21.526220 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.526197    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"kube-root-ca.crt\""
Oct 30 04:18:21.526478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.526442    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"default-dockercfg-zcchk\""
Oct 30 04:18:21.526572 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.526487    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"simulator-w1-6796b1-kserve-self-signed-certs\""
Oct 30 04:18:21.528815 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.528789    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"]
Oct 30 04:18:21.544060 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.544037    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-model-cache\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.544214 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.544136    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce161e2f-5b10-4d42-971e-0d51c68be921-tls-certs\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.544214 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.544191    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-dshm\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.544328 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.544307    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-home\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.544430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.544411    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-tmp-dir\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.544475 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.544459    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97cj\" (UniqueName: \"kubernetes.io/projected/ce161e2f-5b10-4d42-971e-0d51c68be921-kube-api-access-v97cj\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.645454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.645415    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-tmp-dir\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.645454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.645469    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97cj\" (UniqueName: \"kubernetes.io/projected/ce161e2f-5b10-4d42-971e-0d51c68be921-kube-api-access-v97cj\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.645698 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.645554    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-model-cache\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.645698 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.645582    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce161e2f-5b10-4d42-971e-0d51c68be921-tls-certs\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.645698 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.645623    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-dshm\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.645877 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.645706    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-home\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.645970 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.645937    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-tmp-dir\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.646138 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.646117    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-model-cache\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.646237 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.646167    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-home\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.648261 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.648239    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce161e2f-5b10-4d42-971e-0d51c68be921-tls-certs\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.648538 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.648514    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-dshm\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.669780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.669748    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97cj\" (UniqueName: \"kubernetes.io/projected/ce161e2f-5b10-4d42-971e-0d51c68be921-kube-api-access-v97cj\") pod \"simulator-w1-6796b1-kserve-7f95bb55-f69mb\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") " pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:21.802727 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:21.802652    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:22.103189 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:22.103162    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"]
Oct 30 04:18:22.104096 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:18:22.104051    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce161e2f_5b10_4d42_971e_0d51c68be921.slice/crio-07fd00730767e19b5f130763f13e818785b778d312c92c09d68dc5f533d833f4 WatchSource:0}: Error finding container 07fd00730767e19b5f130763f13e818785b778d312c92c09d68dc5f533d833f4: Status 404 returned error can't find the container with id 07fd00730767e19b5f130763f13e818785b778d312c92c09d68dc5f533d833f4
Oct 30 04:18:22.443142 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:22.443112    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb" event={"ID":"ce161e2f-5b10-4d42-971e-0d51c68be921","Type":"ContainerStarted","Data":"6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8"}
Oct 30 04:18:22.443312 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:22.443151    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb" event={"ID":"ce161e2f-5b10-4d42-971e-0d51c68be921","Type":"ContainerStarted","Data":"07fd00730767e19b5f130763f13e818785b778d312c92c09d68dc5f533d833f4"}
Oct 30 04:18:22.485923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:22.485877    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb" podStartSLOduration=1.485861621 podStartE2EDuration="1.485861621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:22.485541726 +0000 UTC m=+1282.226964825" watchObservedRunningTime="2026-10-30 04:18:22.485861621 +0000 UTC m=+1282.227284717"
Oct 30 04:18:22.803247 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:22.803163    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:31.803052 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:31.803015    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:31.817170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:31.817140    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:32.517311 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:32.517280    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 04:18:37.208439 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.208400    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"]
Oct 30 04:18:37.212758 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.212731    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.231806 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.231779    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"]
Oct 30 04:18:37.241911 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.241881    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a2f984\"/\"openshift-service-ca.crt\""
Oct 30 04:18:37.241911 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.241897    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a2f984\"/\"kube-root-ca.crt\""
Oct 30 04:18:37.244418 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.244392    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a2f984\"/\"default-dockercfg-sqvb6\""
Oct 30 04:18:37.244543 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.244430    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a2f984\"/\"premium-w4-a2f984-kserve-self-signed-certs\""
Oct 30 04:18:37.305649 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.305611    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tls-certs\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.305649 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.305651    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-home\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.305878 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.305750    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tmp-dir\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.305878 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.305859    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhrv\" (UniqueName: \"kubernetes.io/projected/3a053e0d-687d-4926-b1d5-cb40e2c769f9-kube-api-access-kfhrv\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.305988 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.305890    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-model-cache\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.305988 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.305960    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-dshm\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.407009 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.406977    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tls-certs\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.407009 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.407015    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-home\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.407287 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.407049    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tmp-dir\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.407287 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.407175    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhrv\" (UniqueName: \"kubernetes.io/projected/3a053e0d-687d-4926-b1d5-cb40e2c769f9-kube-api-access-kfhrv\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.407287 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.407221    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-model-cache\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.407455 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.407377    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-dshm\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.407518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.407498    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-home\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.407608 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.407586    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-model-cache\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.407761 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.407747    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tmp-dir\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.409742 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.409712    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-dshm\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.410324 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.410304    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tls-certs\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.431524 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.431492    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhrv\" (UniqueName: \"kubernetes.io/projected/3a053e0d-687d-4926-b1d5-cb40e2c769f9-kube-api-access-kfhrv\") pod \"premium-w4-a2f984-kserve-68495d4cc6-ltlb2\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") " pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.527328 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.527247    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:37.831167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:37.831140    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"]
Oct 30 04:18:37.831697 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:18:37.831674    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a053e0d_687d_4926_b1d5_cb40e2c769f9.slice/crio-8f07f9360ff2530673efc4ef99b7a55aaba7a9effc6673e21babf586d821f483 WatchSource:0}: Error finding container 8f07f9360ff2530673efc4ef99b7a55aaba7a9effc6673e21babf586d821f483: Status 404 returned error can't find the container with id 8f07f9360ff2530673efc4ef99b7a55aaba7a9effc6673e21babf586d821f483
Oct 30 04:18:38.542298 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:38.542262    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2" event={"ID":"3a053e0d-687d-4926-b1d5-cb40e2c769f9","Type":"ContainerStarted","Data":"5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562"}
Oct 30 04:18:38.542706 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:38.542310    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2" event={"ID":"3a053e0d-687d-4926-b1d5-cb40e2c769f9","Type":"ContainerStarted","Data":"8f07f9360ff2530673efc4ef99b7a55aaba7a9effc6673e21babf586d821f483"}
Oct 30 04:18:38.586808 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:38.586763    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2" podStartSLOduration=1.586748294 podStartE2EDuration="1.586748294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:38.58496582 +0000 UTC m=+1298.326388919" watchObservedRunningTime="2026-10-30 04:18:38.586748294 +0000 UTC m=+1298.328171390"
Oct 30 04:18:39.528366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:39.528333    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:41.751693 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:18:41.751658    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:18:47.528278 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:47.528241    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:47.541154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:47.541126    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:18:47.601554 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:18:47.601526    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 04:19:11.750766 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:19:11.750729    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:19:23.117719 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.117690    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"]
Oct 30 04:19:23.121538 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.121519    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.152996 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.152964    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"]
Oct 30 04:19:23.221746 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.221717    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6796b1-openshift-default-dockercfg-h254p\""
Oct 30 04:19:23.274629 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.274592    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.274812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.274642    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a83356-8ef8-4cc7-a0e9-47015986597f-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.274812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.274720    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.274928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.274817    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.274928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.274838    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.274928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.274859    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.274928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.274881    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f6cs\" (UniqueName: \"kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-kube-api-access-4f6cs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.275167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.274947    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.275167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.275020    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.375664 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.375586    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.375806 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.375693    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.375806 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.375736    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a83356-8ef8-4cc7-a0e9-47015986597f-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.375806 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.375766    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.375806 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.375804    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.376032 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.375831    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.376032 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.375863    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.376032 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.375891    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f6cs\" (UniqueName: \"kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-kube-api-access-4f6cs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.376032 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.375929    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.376032 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.376011    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.376390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.376129    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.376390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.376297    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.376504 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.376470    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.376504 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.376496    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a83356-8ef8-4cc7-a0e9-47015986597f-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.378228 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.378208    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.378782 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.378768    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.405190 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.405169    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.409837 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.409811    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f6cs\" (UniqueName: \"kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-kube-api-access-4f6cs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.436992 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.436970    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:23.758216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.758189    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"]
Oct 30 04:19:23.758585 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:19:23.758561    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a83356_8ef8_4cc7_a0e9_47015986597f.slice/crio-27d0c74e0f30250e8c8206dac393f15727dd470c51cf075a0c06ca939c3d0d1a WatchSource:0}: Error finding container 27d0c74e0f30250e8c8206dac393f15727dd470c51cf075a0c06ca939c3d0d1a: Status 404 returned error can't find the container with id 27d0c74e0f30250e8c8206dac393f15727dd470c51cf075a0c06ca939c3d0d1a
Oct 30 04:19:23.760856 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.760822    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:19:23.760970 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.760897    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:19:23.760970 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.760942    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:19:23.780842 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:23.780815    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs" event={"ID":"11a83356-8ef8-4cc7-a0e9-47015986597f","Type":"ContainerStarted","Data":"27d0c74e0f30250e8c8206dac393f15727dd470c51cf075a0c06ca939c3d0d1a"}
Oct 30 04:19:24.258443 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.258392    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"]
Oct 30 04:19:24.265995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.264801    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:24.286500 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.286467    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"]
Oct 30 04:19:24.292216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.292190    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:19:24.292451 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.292422    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:19:24.387916 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.387875    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96ba317c-sj5q9\" (UID: \"16d14cae-d09f-4340-9ede-2aa115b7d59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:24.388117 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.387973    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2zkl\" (UniqueName: \"kubernetes.io/projected/16d14cae-d09f-4340-9ede-2aa115b7d59d-kube-api-access-v2zkl\") pod \"maas-api-revoke-sub-96ba317c-sj5q9\" (UID: \"16d14cae-d09f-4340-9ede-2aa115b7d59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:24.388182 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.388157    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96ba317c-sj5q9\" (UID: \"16d14cae-d09f-4340-9ede-2aa115b7d59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:24.489764 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.489674    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zkl\" (UniqueName: \"kubernetes.io/projected/16d14cae-d09f-4340-9ede-2aa115b7d59d-kube-api-access-v2zkl\") pod \"maas-api-revoke-sub-96ba317c-sj5q9\" (UID: \"16d14cae-d09f-4340-9ede-2aa115b7d59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:24.489973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.489871    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96ba317c-sj5q9\" (UID: \"16d14cae-d09f-4340-9ede-2aa115b7d59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:24.489973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.489965    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96ba317c-sj5q9\" (UID: \"16d14cae-d09f-4340-9ede-2aa115b7d59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:24.490675 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.490647    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96ba317c-sj5q9\" (UID: \"16d14cae-d09f-4340-9ede-2aa115b7d59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:24.492515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.492495    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96ba317c-sj5q9\" (UID: \"16d14cae-d09f-4340-9ede-2aa115b7d59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:24.514195 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.514132    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zkl\" (UniqueName: \"kubernetes.io/projected/16d14cae-d09f-4340-9ede-2aa115b7d59d-kube-api-access-v2zkl\") pod \"maas-api-revoke-sub-96ba317c-sj5q9\" (UID: \"16d14cae-d09f-4340-9ede-2aa115b7d59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:24.604056 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.604023    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:24.788784 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.788684    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs" event={"ID":"11a83356-8ef8-4cc7-a0e9-47015986597f","Type":"ContainerStarted","Data":"04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4"}
Oct 30 04:19:24.837271 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.837220    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs" podStartSLOduration=1.837204764 podStartE2EDuration="1.837204764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:24.834374793 +0000 UTC m=+1344.575797881" watchObservedRunningTime="2026-10-30 04:19:24.837204764 +0000 UTC m=+1344.578627860"
Oct 30 04:19:24.949512 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.949472    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"]
Oct 30 04:19:24.949512 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:19:24.949508    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d14cae_d09f_4340_9ede_2aa115b7d59d.slice/crio-642bda42d280c1321f29fcab7d5b5417aaaa96a918d9bae97cf56da0afe89160 WatchSource:0}: Error finding container 642bda42d280c1321f29fcab7d5b5417aaaa96a918d9bae97cf56da0afe89160: Status 404 returned error can't find the container with id 642bda42d280c1321f29fcab7d5b5417aaaa96a918d9bae97cf56da0afe89160
Oct 30 04:19:24.951453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:24.951438    5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:19:25.437972 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:25.437935    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:25.439502 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:25.439478    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:19:25.439563 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:25.439532    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs" podUID="11a83356-8ef8-4cc7-a0e9-47015986597f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 04:19:25.803622 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:25.803533    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9" event={"ID":"16d14cae-d09f-4340-9ede-2aa115b7d59d","Type":"ContainerStarted","Data":"642bda42d280c1321f29fcab7d5b5417aaaa96a918d9bae97cf56da0afe89160"}
Oct 30 04:19:26.438096 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:26.438040    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:19:26.438761 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:26.438118    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs" podUID="11a83356-8ef8-4cc7-a0e9-47015986597f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 04:19:27.438102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:27.438046    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:19:27.438522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:27.438120    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs" podUID="11a83356-8ef8-4cc7-a0e9-47015986597f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 04:19:28.437576 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:28.437542    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:19:28.437735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:28.437595    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs" podUID="11a83356-8ef8-4cc7-a0e9-47015986597f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 04:19:28.822460 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:28.822423    5072 generic.go:356] "Generic (PLEG): container finished" podID="16d14cae-d09f-4340-9ede-2aa115b7d59d" containerID="307904a5a078e22664ab5820298f02346ebb1d14baa7c62ddcc9109869cff462" exitCode=0
Oct 30 04:19:28.822809 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:28.822498    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9" event={"ID":"16d14cae-d09f-4340-9ede-2aa115b7d59d","Type":"ContainerDied","Data":"307904a5a078e22664ab5820298f02346ebb1d14baa7c62ddcc9109869cff462"}
Oct 30 04:19:29.438004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:29.437972    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:19:29.438209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:29.438024    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs" podUID="11a83356-8ef8-4cc7-a0e9-47015986597f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 04:19:29.993703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:29.993678    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:30.042219 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.042183    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-xks-ca\") pod \"16d14cae-d09f-4340-9ede-2aa115b7d59d\" (UID: \"16d14cae-d09f-4340-9ede-2aa115b7d59d\") "
Oct 30 04:19:30.042413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.042238    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d14cae-d09f-4340-9ede-2aa115b7d59d-kube-api-access-v2zkl\" (UniqueName: \"kubernetes.io/projected/16d14cae-d09f-4340-9ede-2aa115b7d59d-kube-api-access-v2zkl\") pod \"16d14cae-d09f-4340-9ede-2aa115b7d59d\" (UID: \"16d14cae-d09f-4340-9ede-2aa115b7d59d\") "
Oct 30 04:19:30.042413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.042353    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-openshift-ca\") pod \"16d14cae-d09f-4340-9ede-2aa115b7d59d\" (UID: \"16d14cae-d09f-4340-9ede-2aa115b7d59d\") "
Oct 30 04:19:30.043138 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.043110    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-openshift-ca" pod "16d14cae-d09f-4340-9ede-2aa115b7d59d" (UID: "16d14cae-d09f-4340-9ede-2aa115b7d59d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:19:30.045424 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.045396    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d14cae-d09f-4340-9ede-2aa115b7d59d-kube-api-access-v2zkl" pod "16d14cae-d09f-4340-9ede-2aa115b7d59d" (UID: "16d14cae-d09f-4340-9ede-2aa115b7d59d"). InnerVolumeSpecName "kube-api-access-v2zkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:19:30.045509 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.045470    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-xks-ca" pod "16d14cae-d09f-4340-9ede-2aa115b7d59d" (UID: "16d14cae-d09f-4340-9ede-2aa115b7d59d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:30.144217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.144142    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:19:30.144217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.144175    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d14cae-d09f-4340-9ede-2aa115b7d59d-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:19:30.144217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.144186    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2zkl\" (UniqueName: \"kubernetes.io/projected/16d14cae-d09f-4340-9ede-2aa115b7d59d-kube-api-access-v2zkl\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:19:30.526323 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.526291    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:30.526688 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.526668    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:30.527378 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.527359    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:19:30.836730 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.836701    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"
Oct 30 04:19:30.836901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.836740    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9" event={"ID":"16d14cae-d09f-4340-9ede-2aa115b7d59d","Type":"ContainerDied","Data":"642bda42d280c1321f29fcab7d5b5417aaaa96a918d9bae97cf56da0afe89160"}
Oct 30 04:19:30.836901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:30.836780    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="642bda42d280c1321f29fcab7d5b5417aaaa96a918d9bae97cf56da0afe89160"
Oct 30 04:19:35.664068 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.664034    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"]
Oct 30 04:19:35.664595 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.664580    5072 cpu_manager.go:414] "containerMap: removing container" podUID="16d14cae-d09f-4340-9ede-2aa115b7d59d" containerName="revoke-keys"
Oct 30 04:19:35.664649 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.664595    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d14cae-d09f-4340-9ede-2aa115b7d59d" containerName="revoke-keys"
Oct 30 04:19:35.664704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.664695    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d14cae-d09f-4340-9ede-2aa115b7d59d" containerName="revoke-keys"
Oct 30 04:19:35.668136 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.668117    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.683786 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.683768    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5d74b9-openshift-default-dockercfg-d8777\""
Oct 30 04:19:35.713709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.713672    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxpp\" (UniqueName: \"kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-kube-api-access-zkxpp\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.713956 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.713936    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.714069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.714054    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.714674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.714638    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.714817 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.714796    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5e829f-493e-46f1-84de-c77c944d1108-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.714888 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.714850    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5e829f-493e-46f1-84de-c77c944d1108-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.714948 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.714897    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.715050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.715031    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.715302 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.715278    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.716749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.716717    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"]
Oct 30 04:19:35.816851 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.816823    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5e829f-493e-46f1-84de-c77c944d1108-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.816991 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.816867    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.816991 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.816953    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.817132 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.816996    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.817132 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.817034    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxpp\" (UniqueName: \"kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-kube-api-access-zkxpp\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.817132 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.817088    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.817317 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.817288    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.817383 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.817326    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.817383 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.817349    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.817478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.817406    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.817478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.817429    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.817478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.817467    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5e829f-493e-46f1-84de-c77c944d1108-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.817724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.817703    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.817805 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.817756    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5e829f-493e-46f1-84de-c77c944d1108-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.819545 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.819520    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.820004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.819985    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5e829f-493e-46f1-84de-c77c944d1108-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.841223 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.841203    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.841404 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.841388    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxpp\" (UniqueName: \"kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-kube-api-access-zkxpp\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:35.981132 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:35.981098    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:36.309881 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:36.309855    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"]
Oct 30 04:19:36.310027 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:19:36.310007    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5e829f_493e_46f1_84de_c77c944d1108.slice/crio-8db1a541434e4fcf14d5566e93c0c1992315d7cc3f74026ea952b0bd68ab109a WatchSource:0}: Error finding container 8db1a541434e4fcf14d5566e93c0c1992315d7cc3f74026ea952b0bd68ab109a: Status 404 returned error can't find the container with id 8db1a541434e4fcf14d5566e93c0c1992315d7cc3f74026ea952b0bd68ab109a
Oct 30 04:19:36.312398 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:36.312371    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:19:36.312472 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:36.312450    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:19:36.312519 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:36.312481    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:19:36.880099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:36.879494    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7" event={"ID":"1e5e829f-493e-46f1-84de-c77c944d1108","Type":"ContainerStarted","Data":"7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614"}
Oct 30 04:19:36.880099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:36.879623    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7" event={"ID":"1e5e829f-493e-46f1-84de-c77c944d1108","Type":"ContainerStarted","Data":"8db1a541434e4fcf14d5566e93c0c1992315d7cc3f74026ea952b0bd68ab109a"}
Oct 30 04:19:36.942577 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:36.942530    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7" podStartSLOduration=1.942514943 podStartE2EDuration="1.942514943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:36.940457959 +0000 UTC m=+1356.681881055" watchObservedRunningTime="2026-10-30 04:19:36.942514943 +0000 UTC m=+1356.683938039"
Oct 30 04:19:36.982060 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:36.982010    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:37.983893 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:37.983848    5072 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:19:37.984348 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:37.983917    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7" podUID="1e5e829f-493e-46f1-84de-c77c944d1108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 04:19:38.982663 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:38.982627    5072 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:19:38.982934 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:38.982903    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7" podUID="1e5e829f-493e-46f1-84de-c77c944d1108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 04:19:39.982496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:39.982455    5072 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:19:39.983058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:39.982511    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7" podUID="1e5e829f-493e-46f1-84de-c77c944d1108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 04:19:40.982190 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:40.982148    5072 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:19:40.982419 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:40.982206    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7" podUID="1e5e829f-493e-46f1-84de-c77c944d1108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 04:19:41.751555 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:19:41.751517    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:19:41.985984 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:41.985518    5072 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:19:41.985984 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:41.985581    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7" podUID="1e5e829f-493e-46f1-84de-c77c944d1108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 04:19:42.628703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.628669    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"]
Oct 30 04:19:42.633069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.633047    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.665590 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.665553    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-36bbac-openshift-default-dockercfg-n6t7k\""
Oct 30 04:19:42.669523 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.669499    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"]
Oct 30 04:19:42.708981 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.708951    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.709177 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.708989    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.709177 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.709010    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vtcj\" (UniqueName: \"kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-kube-api-access-2vtcj\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.709307 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.709213    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.709307 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.709264    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5255247-1989-4d28-a995-f5256f07e2db-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.709387 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.709326    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5255247-1989-4d28-a995-f5256f07e2db-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.709387 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.709355    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.709464 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.709420    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.709464 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.709453    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.810435 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.810396    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.810435 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.810440    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.810944 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.810458    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vtcj\" (UniqueName: \"kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-kube-api-access-2vtcj\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.810944 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.810495    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.810944 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.810619    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5255247-1989-4d28-a995-f5256f07e2db-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.810944 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.810671    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5255247-1989-4d28-a995-f5256f07e2db-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.810944 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.810703    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.810944 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.810757    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.810944 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.810808    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.811307 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.811189    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.811307 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.811204    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.811307 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.811271    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.811477 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.811459    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5255247-1989-4d28-a995-f5256f07e2db-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.811519 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.811465    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.813049 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.813024    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.813736 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.813716    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5255247-1989-4d28-a995-f5256f07e2db-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.834665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.834635    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.834816 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.834795    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vtcj\" (UniqueName: \"kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-kube-api-access-2vtcj\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.953174 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.953143    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:42.986885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.986857    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:42.987574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.987552    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:42.988135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:42.988071    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:19:43.290541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:43.290514    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"]
Oct 30 04:19:43.290659 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:19:43.290589    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5255247_1989_4d28_a995_f5256f07e2db.slice/crio-99d8c074b3eab23e5ed3670e735d4f260c42170515ee033df692551cda469d10 WatchSource:0}: Error finding container 99d8c074b3eab23e5ed3670e735d4f260c42170515ee033df692551cda469d10: Status 404 returned error can't find the container with id 99d8c074b3eab23e5ed3670e735d4f260c42170515ee033df692551cda469d10
Oct 30 04:19:43.293001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:43.292970    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:19:43.293068 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:43.293045    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:19:43.293138 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:43.293104    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:19:43.925867 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:43.925829    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q" event={"ID":"e5255247-1989-4d28-a995-f5256f07e2db","Type":"ContainerStarted","Data":"091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761"}
Oct 30 04:19:43.926353 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:43.925892    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q" event={"ID":"e5255247-1989-4d28-a995-f5256f07e2db","Type":"ContainerStarted","Data":"99d8c074b3eab23e5ed3670e735d4f260c42170515ee033df692551cda469d10"}
Oct 30 04:19:43.954061 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:43.954037    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:44.954978 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:44.954946    5072 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 04:19:44.955450 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:44.955000    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q" podUID="e5255247-1989-4d28-a995-f5256f07e2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 04:19:45.953921 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:45.953880    5072 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 04:19:45.954135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:45.953931    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q" podUID="e5255247-1989-4d28-a995-f5256f07e2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 04:19:46.953778 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:46.953745    5072 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 04:19:46.954214 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:46.953797    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q" podUID="e5255247-1989-4d28-a995-f5256f07e2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 04:19:47.954006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:47.953974    5072 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 04:19:47.954424 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:47.954028    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q" podUID="e5255247-1989-4d28-a995-f5256f07e2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 04:19:48.953449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:48.953421    5072 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 04:19:48.953628 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:48.953471    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q" podUID="e5255247-1989-4d28-a995-f5256f07e2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 04:19:49.957568 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:49.957533    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:49.958042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:49.958015    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:49.959152 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:49.959133    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 04:19:50.017284 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:50.017230    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q" podStartSLOduration=8.017214847 podStartE2EDuration="8.017214847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:43.983131126 +0000 UTC m=+1363.724554223" watchObservedRunningTime="2026-10-30 04:19:50.017214847 +0000 UTC m=+1369.758638018"
Oct 30 04:19:51.169108 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.169049    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"]
Oct 30 04:19:51.173317 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.173291    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.188798 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.188775    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b4708a-openshift-default-dockercfg-t5rn6\""
Oct 30 04:19:51.204475 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.204440    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"]
Oct 30 04:19:51.237648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.237615    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.237819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.237664    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7f8\" (UniqueName: \"kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-kube-api-access-ch7f8\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.237819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.237685    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.237819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.237772    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.237819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.237792    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.237979 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.237836    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.237979 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.237895    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53c7a3aa-110a-4656-9b44-c4a1cd611042-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.238042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.237977    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.238042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.238014    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.339216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.339181    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.339382 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.339237    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.339382 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.339304    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.339382 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.339341    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7f8\" (UniqueName: \"kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-kube-api-access-ch7f8\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.339382 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.339359    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.339382 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.339379    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.339670 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.339394    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.339670 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.339443    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.339670 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.339472    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53c7a3aa-110a-4656-9b44-c4a1cd611042-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.339843 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.339713    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.339843 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.339766    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.340017 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.339989    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.340188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.340018    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.340188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.340170    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53c7a3aa-110a-4656-9b44-c4a1cd611042-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.342149 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.342122    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.342388 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.342371    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.363537 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.363514    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7f8\" (UniqueName: \"kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-kube-api-access-ch7f8\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.376090 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.376050    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.487835 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.487802    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:51.806823 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.806796    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"]
Oct 30 04:19:51.807300 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:19:51.807278    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c7a3aa_110a_4656_9b44_c4a1cd611042.slice/crio-89a88444bb76afa1b02760fdcd6bdb85ac6949ac3000eff94dc56d3d58ae1d04 WatchSource:0}: Error finding container 89a88444bb76afa1b02760fdcd6bdb85ac6949ac3000eff94dc56d3d58ae1d04: Status 404 returned error can't find the container with id 89a88444bb76afa1b02760fdcd6bdb85ac6949ac3000eff94dc56d3d58ae1d04
Oct 30 04:19:51.809850 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.809820    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:19:51.809952 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.809895    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:19:51.809952 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.809941    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:19:51.974173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.974130    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" event={"ID":"53c7a3aa-110a-4656-9b44-c4a1cd611042","Type":"ContainerStarted","Data":"236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5"}
Oct 30 04:19:51.974347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:51.974199    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" event={"ID":"53c7a3aa-110a-4656-9b44-c4a1cd611042","Type":"ContainerStarted","Data":"89a88444bb76afa1b02760fdcd6bdb85ac6949ac3000eff94dc56d3d58ae1d04"}
Oct 30 04:19:52.021392 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:52.021330    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" podStartSLOduration=1.021314095 podStartE2EDuration="1.021314095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:52.019586049 +0000 UTC m=+1371.761009181" watchObservedRunningTime="2026-10-30 04:19:52.021314095 +0000 UTC m=+1371.762737192"
Oct 30 04:19:52.488375 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:52.488337    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:52.489850 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:52.489823    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 04:19:52.489970 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:52.489885    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" podUID="53c7a3aa-110a-4656-9b44-c4a1cd611042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 04:19:53.488664 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:53.488625    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 04:19:53.489223 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:53.488685    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" podUID="53c7a3aa-110a-4656-9b44-c4a1cd611042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 04:19:54.488314 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:54.488273    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 04:19:54.488525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:54.488326    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" podUID="53c7a3aa-110a-4656-9b44-c4a1cd611042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 04:19:55.488709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:55.488661    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 04:19:55.489164 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:55.488749    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" podUID="53c7a3aa-110a-4656-9b44-c4a1cd611042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 04:19:56.488637 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:56.488598    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 04:19:56.488863 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:56.488661    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" podUID="53c7a3aa-110a-4656-9b44-c4a1cd611042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 04:19:57.488230 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:57.488196    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 04:19:57.488419 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:57.488268    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" podUID="53c7a3aa-110a-4656-9b44-c4a1cd611042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 04:19:59.109517 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:59.109484    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:59.110033 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:59.110010    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:19:59.111185 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:19:59.111156    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:20:11.752022 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:20:11.751938    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:20:21.258743 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.258707    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"]
Oct 30 04:20:21.263429 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.263403    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.280949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.280891    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b4708a\"/\"openshift-service-ca.crt\""
Oct 30 04:20:21.284625 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.284599    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"]
Oct 30 04:20:21.293422 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.293393    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b4708a\"/\"kube-root-ca.crt\""
Oct 30 04:20:21.293422 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.293393    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b4708a\"/\"default-dockercfg-6p4tv\""
Oct 30 04:20:21.293591 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.293457    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b4708a\"/\"distinct-w3-b4708a-kserve-self-signed-certs\""
Oct 30 04:20:21.360904 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.360873    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tmp-dir\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.361122 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.360932    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-model-cache\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.361122 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.361027    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-dshm\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.361201 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.361129    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgrfl\" (UniqueName: \"kubernetes.io/projected/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-kube-api-access-wgrfl\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.361244 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.361226    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-home\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.361324 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.361308    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tls-certs\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.464102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.462749    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-home\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.464102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.462888    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tls-certs\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.464102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.463012    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tmp-dir\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.464102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.463065    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-model-cache\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.474108 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.466012    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-dshm\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.474108 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.463869    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-home\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.474108 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.465896    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-model-cache\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.474108 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.465682    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tmp-dir\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.474108 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.466094    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgrfl\" (UniqueName: \"kubernetes.io/projected/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-kube-api-access-wgrfl\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.474108 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.470321    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-dshm\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.474108 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.474011    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tls-certs\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.493248 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.493215    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgrfl\" (UniqueName: \"kubernetes.io/projected/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-kube-api-access-wgrfl\") pod \"distinct-w3-b4708a-kserve-775fdcd64-f6szl\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") " pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.580027 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.579929    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:21.881978 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:21.881955    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"]
Oct 30 04:20:21.882545 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:20:21.882521    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ff7e01a_1fb4_44e9_b78d_0f59b26c4657.slice/crio-b1921ff84c10b7610f2972f1eaba07fc799e24d2e5b887eadec646dafbbd0d7d WatchSource:0}: Error finding container b1921ff84c10b7610f2972f1eaba07fc799e24d2e5b887eadec646dafbbd0d7d: Status 404 returned error can't find the container with id b1921ff84c10b7610f2972f1eaba07fc799e24d2e5b887eadec646dafbbd0d7d
Oct 30 04:20:22.172190 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:22.172094    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl" event={"ID":"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657","Type":"ContainerStarted","Data":"4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80"}
Oct 30 04:20:22.172190 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:22.172148    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl" event={"ID":"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657","Type":"ContainerStarted","Data":"b1921ff84c10b7610f2972f1eaba07fc799e24d2e5b887eadec646dafbbd0d7d"}
Oct 30 04:20:22.217299 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:22.217253    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl" podStartSLOduration=1.217239553 podStartE2EDuration="1.217239553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:22.215328904 +0000 UTC m=+1401.956752001" watchObservedRunningTime="2026-10-30 04:20:22.217239553 +0000 UTC m=+1401.958662660"
Oct 30 04:20:22.580436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:22.580409    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:31.580914 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:31.580877    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:31.594300 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:31.594275    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:32.248672 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:32.248644    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 04:20:35.842166 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.842134    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"]
Oct 30 04:20:35.846488 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.846469    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.847889 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.847864    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86w4l\" (UniqueName: \"kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-kube-api-access-86w4l\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.848027 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.847899    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.848027 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.847919    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b735ef67-f82e-44b6-9306-fdad4e8ef244-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.848027 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.847989    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.848216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.848094    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.848216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.848151    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.848216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.848172    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.848376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.848225    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.848376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.848256    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.877220 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.877188    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"]
Oct 30 04:20:35.949198 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949164    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86w4l\" (UniqueName: \"kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-kube-api-access-86w4l\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.949380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949203    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.949380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949237    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b735ef67-f82e-44b6-9306-fdad4e8ef244-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.949380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949277    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.949380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949301    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.949380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949348    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.949380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949375    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.949708 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949421    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.949708 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949479    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.949708 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949615    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.949708 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949682    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.949903 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949758    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.949903 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.949884    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.950140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.950110    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b735ef67-f82e-44b6-9306-fdad4e8ef244-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.952046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.952021    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.952314 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.952297    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.973594 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.973573    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:35.973855 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:35.973835    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86w4l\" (UniqueName: \"kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-kube-api-access-86w4l\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:36.163196 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:36.163119    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:36.484980 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:36.484952    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"]
Oct 30 04:20:36.485769 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:20:36.485742    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb735ef67_f82e_44b6_9306_fdad4e8ef244.slice/crio-7fdf3dfcf53dc13f3c4a52abf16851fd45caa1e2ce5dae7e2ed57142b45087b5 WatchSource:0}: Error finding container 7fdf3dfcf53dc13f3c4a52abf16851fd45caa1e2ce5dae7e2ed57142b45087b5: Status 404 returned error can't find the container with id 7fdf3dfcf53dc13f3c4a52abf16851fd45caa1e2ce5dae7e2ed57142b45087b5
Oct 30 04:20:36.488069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:36.488040    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:20:36.488155 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:36.488136    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:20:36.488192 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:36.488168    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:20:37.285854 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:37.285821    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" event={"ID":"b735ef67-f82e-44b6-9306-fdad4e8ef244","Type":"ContainerStarted","Data":"785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948"}
Oct 30 04:20:37.286366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:37.285878    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" event={"ID":"b735ef67-f82e-44b6-9306-fdad4e8ef244","Type":"ContainerStarted","Data":"7fdf3dfcf53dc13f3c4a52abf16851fd45caa1e2ce5dae7e2ed57142b45087b5"}
Oct 30 04:20:37.347869 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:37.347822    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" podStartSLOduration=2.347807162 podStartE2EDuration="2.347807162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:37.345127761 +0000 UTC m=+1417.086550861" watchObservedRunningTime="2026-10-30 04:20:37.347807162 +0000 UTC m=+1417.089230258"
Oct 30 04:20:38.166173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.166145    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:38.166499 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.166478    5072 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 04:20:38.166646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.166618    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" podUID="b735ef67-f82e-44b6-9306-fdad4e8ef244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 04:20:38.329487 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.329452    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"]
Oct 30 04:20:38.333641 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.333624    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.395964 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.393570    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"]
Oct 30 04:20:38.405357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.404604    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.405357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.404802    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.405357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.404827    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.405357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.404855    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.405357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.404883    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.405357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.404993    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.405357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.405046    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.405357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.405066    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqlmr\" (UniqueName: \"kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-kube-api-access-tqlmr\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.405357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.405122    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.505804 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.505755    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.506024 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.505854    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.506024 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.505873    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.506024 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.505908    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.506024 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.505954    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.506300 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.506142    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.506300 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.506157    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.506999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.506300    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.506999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.506344    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqlmr\" (UniqueName: \"kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-kube-api-access-tqlmr\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.506999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.506391    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.506999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.506464    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.506999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.506561    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.506999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.506662    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.507479 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.507113    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.509104 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.509059    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.509366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.509350    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.530094 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.530043    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.530943 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.530919    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqlmr\" (UniqueName: \"kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-kube-api-access-tqlmr\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.650215 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.650177    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:38.997268 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:20:38.997236    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde013201_a8b0_4d48_b0f3_d17fc1e9d952.slice/crio-bcb52d21fef1f3c33a9cc6a82ba96360c6c70e847a2b91fd529b7b77614b2fb6 WatchSource:0}: Error finding container bcb52d21fef1f3c33a9cc6a82ba96360c6c70e847a2b91fd529b7b77614b2fb6: Status 404 returned error can't find the container with id bcb52d21fef1f3c33a9cc6a82ba96360c6c70e847a2b91fd529b7b77614b2fb6
Oct 30 04:20:38.997397 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:38.997376    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"]
Oct 30 04:20:39.000400 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:39.000231    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:20:39.000400 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:39.000309    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:20:39.001052 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:39.000648    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:20:39.173533 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:39.173499    5072 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 04:20:39.173799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:39.173770    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" podUID="b735ef67-f82e-44b6-9306-fdad4e8ef244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 04:20:39.311671 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:39.311576    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22" event={"ID":"de013201-a8b0-4d48-b0f3-d17fc1e9d952","Type":"ContainerStarted","Data":"30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a"}
Oct 30 04:20:39.311671 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:39.311637    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22" event={"ID":"de013201-a8b0-4d48-b0f3-d17fc1e9d952","Type":"ContainerStarted","Data":"bcb52d21fef1f3c33a9cc6a82ba96360c6c70e847a2b91fd529b7b77614b2fb6"}
Oct 30 04:20:39.386497 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:39.386431    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22" podStartSLOduration=1.3864080140000001 podStartE2EDuration="1.386408014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:39.370965784 +0000 UTC m=+1419.112388882" watchObservedRunningTime="2026-10-30 04:20:39.386408014 +0000 UTC m=+1419.127831112"
Oct 30 04:20:39.650602 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:39.650513    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:40.164521 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:40.164473    5072 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 04:20:40.164805 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:40.164776    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" podUID="b735ef67-f82e-44b6-9306-fdad4e8ef244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 04:20:40.652798 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:40.652766    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 04:20:40.653186 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:40.652843    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22" podUID="de013201-a8b0-4d48-b0f3-d17fc1e9d952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 04:20:41.171103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:41.167470    5072 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 04:20:41.171103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:41.167530    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" podUID="b735ef67-f82e-44b6-9306-fdad4e8ef244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 04:20:41.650556 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:41.650528    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 04:20:41.650718 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:41.650586    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22" podUID="de013201-a8b0-4d48-b0f3-d17fc1e9d952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 04:20:41.753593 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:20:41.753546    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:20:42.164509 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:42.164464    5072 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 04:20:42.164783 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:42.164751    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" podUID="b735ef67-f82e-44b6-9306-fdad4e8ef244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 04:20:42.650559 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:42.650524    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 04:20:42.650799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:42.650579    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22" podUID="de013201-a8b0-4d48-b0f3-d17fc1e9d952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 04:20:43.173102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:43.166443    5072 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 04:20:43.173102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:43.166517    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" podUID="b735ef67-f82e-44b6-9306-fdad4e8ef244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 04:20:43.650821 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:43.650782    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 04:20:43.650990 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:43.650843    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22" podUID="de013201-a8b0-4d48-b0f3-d17fc1e9d952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 04:20:44.180427 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:44.180395    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:44.181672 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:44.181649    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:44.182705 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:44.182688    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 04:20:44.650698 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:44.650665    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 04:20:44.650884 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:44.650714    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22" podUID="de013201-a8b0-4d48-b0f3-d17fc1e9d952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 04:20:45.651566 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:45.651534    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 04:20:45.652115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:45.651585    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22" podUID="de013201-a8b0-4d48-b0f3-d17fc1e9d952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 04:20:47.479183 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:47.479148    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:47.479609 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:47.479563    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:47.480632 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:47.480606    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:20:51.343868 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.343837    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"]
Oct 30 04:20:51.347996 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.347971    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.374861 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.374826    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"]
Oct 30 04:20:51.492487 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.492453    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.492709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.492675    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.492771 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.492716    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.492771 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.492748    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.492871 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.492850    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.492927 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.492876    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkhjj\" (UniqueName: \"kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-kube-api-access-fkhjj\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.492927 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.492909    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535f2452-a76b-4861-87b1-0a278f03f35a-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.493025 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.493006    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.493168 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.493154    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535f2452-a76b-4861-87b1-0a278f03f35a-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.594593 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.594503    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.594593 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.594545    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.594593 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.594571    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.594873 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.594614    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.594873 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.594645    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkhjj\" (UniqueName: \"kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-kube-api-access-fkhjj\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.594873 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.594680    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535f2452-a76b-4861-87b1-0a278f03f35a-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.594873 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.594711    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.594873 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.594788    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535f2452-a76b-4861-87b1-0a278f03f35a-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.594873 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.594822    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.595206 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.594933    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.595206 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.594951    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.595206 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.595160    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.595527 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.595229    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.595728 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.595707    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535f2452-a76b-4861-87b1-0a278f03f35a-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.597134 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.597113    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.597909 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.597887    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535f2452-a76b-4861-87b1-0a278f03f35a-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.621445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.621421    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.621743 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.621724    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkhjj\" (UniqueName: \"kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-kube-api-access-fkhjj\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.663975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.663948    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:51.999938 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:51.999890    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"]
Oct 30 04:20:52.002631 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:20:52.002568    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535f2452_a76b_4861_87b1_0a278f03f35a.slice/crio-fac7956f1d881a43504127692bd5acf1e2ebfd37f0fd336b008a803be527df4c WatchSource:0}: Error finding container fac7956f1d881a43504127692bd5acf1e2ebfd37f0fd336b008a803be527df4c: Status 404 returned error can't find the container with id fac7956f1d881a43504127692bd5acf1e2ebfd37f0fd336b008a803be527df4c
Oct 30 04:20:52.005370 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:52.005334    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:20:52.005579 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:52.005554    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:20:52.005663 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:52.005608    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:20:52.399064 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:52.398974    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd" event={"ID":"535f2452-a76b-4861-87b1-0a278f03f35a","Type":"ContainerStarted","Data":"3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742"}
Oct 30 04:20:52.399064 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:52.399037    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd" event={"ID":"535f2452-a76b-4861-87b1-0a278f03f35a","Type":"ContainerStarted","Data":"fac7956f1d881a43504127692bd5acf1e2ebfd37f0fd336b008a803be527df4c"}
Oct 30 04:20:52.448675 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:52.448617    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd" podStartSLOduration=1.448599131 podStartE2EDuration="1.448599131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:52.444924789 +0000 UTC m=+1432.186347891" watchObservedRunningTime="2026-10-30 04:20:52.448599131 +0000 UTC m=+1432.190022228"
Oct 30 04:20:52.664184 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:52.664109    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:53.679807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:53.679458    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 04:20:53.679807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:53.679525    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd" podUID="535f2452-a76b-4861-87b1-0a278f03f35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 04:20:54.680115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:54.676398    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 04:20:54.680115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:54.676456    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd" podUID="535f2452-a76b-4861-87b1-0a278f03f35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 04:20:55.673469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:55.673432    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 04:20:55.673852 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:55.673816    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd" podUID="535f2452-a76b-4861-87b1-0a278f03f35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 04:20:56.666103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:56.664439    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 04:20:56.666103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:56.664496    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd" podUID="535f2452-a76b-4861-87b1-0a278f03f35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 04:20:57.681100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:57.667128    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 04:20:57.681100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:57.667203    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd" podUID="535f2452-a76b-4861-87b1-0a278f03f35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 04:20:58.665100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:58.665042    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 04:20:58.665267 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:58.665133    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd" podUID="535f2452-a76b-4861-87b1-0a278f03f35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 04:20:59.670468 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:59.670435    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:59.670847 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:59.670711    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:20:59.671521 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:20:59.671504    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:21:11.751785 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:21:11.751746    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:21:23.463491 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.463453    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"]
Oct 30 04:21:23.467765 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.467750    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.490565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.490539    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"]
Oct 30 04:21:23.576352 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.576320    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.576352 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.576353    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.576605 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.576533    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1b7028-ef70-47a1-a530-275f970c88aa-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.576665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.576622    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.576731 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.576713    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.576788 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.576743    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frrw4\" (UniqueName: \"kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-kube-api-access-frrw4\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.576839 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.576791    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.576839 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.576812    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.576839 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.576830    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.677981 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.677953    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1b7028-ef70-47a1-a530-275f970c88aa-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.677988    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.678031    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.678046    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrw4\" (UniqueName: \"kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-kube-api-access-frrw4\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.678067    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.678105    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.678134    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678488 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.678256    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678488 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.678284    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678488 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.678453    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678645 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.678551    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678645 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.678589    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678747 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.678713    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.678863 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.678807    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1b7028-ef70-47a1-a530-275f970c88aa-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.680653 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.680626    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.681042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.681022    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.704592 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.704564    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.704702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.704688    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrw4\" (UniqueName: \"kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-kube-api-access-frrw4\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:23.782436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:23.782380    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:24.102230 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:24.102204    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"]
Oct 30 04:21:24.104690 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:21:24.104661    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1b7028_ef70_47a1_a530_275f970c88aa.slice/crio-53468e846c72a58a82dfc5fbf599ff262c478c4b64c802e4bae78a3750ad8624 WatchSource:0}: Error finding container 53468e846c72a58a82dfc5fbf599ff262c478c4b64c802e4bae78a3750ad8624: Status 404 returned error can't find the container with id 53468e846c72a58a82dfc5fbf599ff262c478c4b64c802e4bae78a3750ad8624
Oct 30 04:21:24.107006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:24.106972    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:21:24.107106 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:24.107038    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:21:24.107106 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:24.107067    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:21:24.597650 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:24.597613    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb" event={"ID":"5c1b7028-ef70-47a1-a530-275f970c88aa","Type":"ContainerStarted","Data":"e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600"}
Oct 30 04:21:24.598044 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:24.597671    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb" event={"ID":"5c1b7028-ef70-47a1-a530-275f970c88aa","Type":"ContainerStarted","Data":"53468e846c72a58a82dfc5fbf599ff262c478c4b64c802e4bae78a3750ad8624"}
Oct 30 04:21:24.646874 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:24.646828    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb" podStartSLOduration=1.646814278 podStartE2EDuration="1.646814278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:24.644836369 +0000 UTC m=+1464.386259471" watchObservedRunningTime="2026-10-30 04:21:24.646814278 +0000 UTC m=+1464.388237374"
Oct 30 04:21:24.783067 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:24.783022    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:25.783917 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:25.783884    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 04:21:25.784323 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:25.783957    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb" podUID="5c1b7028-ef70-47a1-a530-275f970c88aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 04:21:26.783605 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:26.783570    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 04:21:26.783773 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:26.783643    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb" podUID="5c1b7028-ef70-47a1-a530-275f970c88aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 04:21:27.783304 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:27.783269    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 04:21:27.783726 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:27.783324    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb" podUID="5c1b7028-ef70-47a1-a530-275f970c88aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 04:21:28.783294 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:28.783261    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 04:21:28.783499 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:28.783314    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb" podUID="5c1b7028-ef70-47a1-a530-275f970c88aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 04:21:29.783768 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:29.783729    5072 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 04:21:29.784299 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:29.783792    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb" podUID="5c1b7028-ef70-47a1-a530-275f970c88aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 04:21:30.988308 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:30.988279    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:30.988822 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:30.988772    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:30.992440 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:21:30.992418    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 04:21:41.755836 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:21:41.755720    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:22:00.579236 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:22:00.579201    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:22:00.584212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:22:00.584190    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:22:11.751798 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:22:11.751761    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:22:41.751386 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:22:41.751344    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:23:04.939433 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:04.939389    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"]
Oct 30 04:23:04.940045 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:04.939641    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx" podUID="27f88e7d-21c3-4e59-9665-5cb24a87b5d5" containerName="istio-proxy" containerID="cri-o://ff6d95cfc32516967d9e1396ced5a116b3a0a7f9601d0268e0fa72b9b77aaa4a" gracePeriod=30
Oct 30 04:23:04.953470 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:04.953437    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"]
Oct 30 04:23:04.953754 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:04.953726    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg" podUID="9f813b35-b180-40dd-bdd1-91fffa7e133e" containerName="istio-proxy" containerID="cri-o://d810ba66e358f5b7a5e1259a2b5e9bfa89349bb45b8b97bb8bd2ea4765d181da" gracePeriod=30
Oct 30 04:23:10.236786 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.236757    5072 generic.go:356] "Generic (PLEG): container finished" podID="9f813b35-b180-40dd-bdd1-91fffa7e133e" containerID="d810ba66e358f5b7a5e1259a2b5e9bfa89349bb45b8b97bb8bd2ea4765d181da" exitCode=0
Oct 30 04:23:10.237159 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.236818    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg" event={"ID":"9f813b35-b180-40dd-bdd1-91fffa7e133e","Type":"ContainerDied","Data":"d810ba66e358f5b7a5e1259a2b5e9bfa89349bb45b8b97bb8bd2ea4765d181da"}
Oct 30 04:23:10.237159 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.236873    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg" event={"ID":"9f813b35-b180-40dd-bdd1-91fffa7e133e","Type":"ContainerDied","Data":"96ef6bebdd042b5bfa2c8a9528b84cb7be70ed8e60dfda32a5b4958e1f6eb322"}
Oct 30 04:23:10.237159 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.236900    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96ef6bebdd042b5bfa2c8a9528b84cb7be70ed8e60dfda32a5b4958e1f6eb322"
Oct 30 04:23:10.238591 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.238568    5072 generic.go:356] "Generic (PLEG): container finished" podID="27f88e7d-21c3-4e59-9665-5cb24a87b5d5" containerID="ff6d95cfc32516967d9e1396ced5a116b3a0a7f9601d0268e0fa72b9b77aaa4a" exitCode=0
Oct 30 04:23:10.238704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.238638    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx" event={"ID":"27f88e7d-21c3-4e59-9665-5cb24a87b5d5","Type":"ContainerDied","Data":"ff6d95cfc32516967d9e1396ced5a116b3a0a7f9601d0268e0fa72b9b77aaa4a"}
Oct 30 04:23:10.238704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.238674    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx" event={"ID":"27f88e7d-21c3-4e59-9665-5cb24a87b5d5","Type":"ContainerDied","Data":"9e64b02ee496479dc5659620e79aaf39f655eff35a2f20a9af31964c1a55d37c"}
Oct 30 04:23:10.238704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.238689    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e64b02ee496479dc5659620e79aaf39f655eff35a2f20a9af31964c1a55d37c"
Oct 30 04:23:10.242962 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.242941    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:23:10.246448 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.246428    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:23:10.304413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.304386    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-kube-api-access-tfcbc\" (UniqueName: \"kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-kube-api-access-tfcbc\") pod \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") "
Oct 30 04:23:10.304924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.304454    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-podinfo\") pod \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") "
Oct 30 04:23:10.304924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.304633    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-certs\") pod \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") "
Oct 30 04:23:10.304924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.304705    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-credential-socket\") pod \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") "
Oct 30 04:23:10.304924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.304760    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istiod-ca-cert\") pod \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") "
Oct 30 04:23:10.304924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.304835    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-envoy\") pod \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") "
Oct 30 04:23:10.304924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.304888    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-token\" (UniqueName: \"kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-token\") pod \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") "
Oct 30 04:23:10.304924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.304898    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-certs" pod "27f88e7d-21c3-4e59-9665-5cb24a87b5d5" (UID: "27f88e7d-21c3-4e59-9665-5cb24a87b5d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:10.305386 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.304992    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-socket\") pod \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") "
Oct 30 04:23:10.305386 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.305017    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-data\") pod \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\" (UID: \"27f88e7d-21c3-4e59-9665-5cb24a87b5d5\") "
Oct 30 04:23:10.305606 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.305552    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.305856 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.305830    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istiod-ca-cert" pod "27f88e7d-21c3-4e59-9665-5cb24a87b5d5" (UID: "27f88e7d-21c3-4e59-9665-5cb24a87b5d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:23:10.306335 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.306153    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-credential-socket" pod "27f88e7d-21c3-4e59-9665-5cb24a87b5d5" (UID: "27f88e7d-21c3-4e59-9665-5cb24a87b5d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:10.306335 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.306186    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-socket" pod "27f88e7d-21c3-4e59-9665-5cb24a87b5d5" (UID: "27f88e7d-21c3-4e59-9665-5cb24a87b5d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:10.306335 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.306310    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-data" pod "27f88e7d-21c3-4e59-9665-5cb24a87b5d5" (UID: "27f88e7d-21c3-4e59-9665-5cb24a87b5d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:10.307486 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.307460    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-podinfo" pod "27f88e7d-21c3-4e59-9665-5cb24a87b5d5" (UID: "27f88e7d-21c3-4e59-9665-5cb24a87b5d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:23:10.307684 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.307655    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-envoy" pod "27f88e7d-21c3-4e59-9665-5cb24a87b5d5" (UID: "27f88e7d-21c3-4e59-9665-5cb24a87b5d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:10.307784 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.307738    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-token" pod "27f88e7d-21c3-4e59-9665-5cb24a87b5d5" (UID: "27f88e7d-21c3-4e59-9665-5cb24a87b5d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:23:10.308411 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.308394    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-kube-api-access-tfcbc" pod "27f88e7d-21c3-4e59-9665-5cb24a87b5d5" (UID: "27f88e7d-21c3-4e59-9665-5cb24a87b5d5"). InnerVolumeSpecName "kube-api-access-tfcbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:23:10.405872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.405846    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-certs\") pod \"9f813b35-b180-40dd-bdd1-91fffa7e133e\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") "
Oct 30 04:23:10.406062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.405915    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f813b35-b180-40dd-bdd1-91fffa7e133e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f813b35-b180-40dd-bdd1-91fffa7e133e-istiod-ca-cert\") pod \"9f813b35-b180-40dd-bdd1-91fffa7e133e\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") "
Oct 30 04:23:10.406062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.405933    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-data\") pod \"9f813b35-b180-40dd-bdd1-91fffa7e133e\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") "
Oct 30 04:23:10.406062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.405954    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-credential-socket\") pod \"9f813b35-b180-40dd-bdd1-91fffa7e133e\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") "
Oct 30 04:23:10.406062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.406024    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-socket\") pod \"9f813b35-b180-40dd-bdd1-91fffa7e133e\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") "
Oct 30 04:23:10.406062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.406060    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-podinfo\") pod \"9f813b35-b180-40dd-bdd1-91fffa7e133e\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") "
Oct 30 04:23:10.406388 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.406120    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-token\" (UniqueName: \"kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-token\") pod \"9f813b35-b180-40dd-bdd1-91fffa7e133e\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") "
Oct 30 04:23:10.406709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.406139    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-certs" pod "9f813b35-b180-40dd-bdd1-91fffa7e133e" (UID: "9f813b35-b180-40dd-bdd1-91fffa7e133e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:10.406709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.406289    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-data" pod "9f813b35-b180-40dd-bdd1-91fffa7e133e" (UID: "9f813b35-b180-40dd-bdd1-91fffa7e133e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:10.406709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.406334    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-credential-socket" pod "9f813b35-b180-40dd-bdd1-91fffa7e133e" (UID: "9f813b35-b180-40dd-bdd1-91fffa7e133e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:10.406709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.406371    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f813b35-b180-40dd-bdd1-91fffa7e133e-istiod-ca-cert" pod "9f813b35-b180-40dd-bdd1-91fffa7e133e" (UID: "9f813b35-b180-40dd-bdd1-91fffa7e133e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:23:10.406709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.406374    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-socket" pod "9f813b35-b180-40dd-bdd1-91fffa7e133e" (UID: "9f813b35-b180-40dd-bdd1-91fffa7e133e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:10.407259 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.406159    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-kube-api-access-lzd4z\" (UniqueName: \"kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-kube-api-access-lzd4z\") pod \"9f813b35-b180-40dd-bdd1-91fffa7e133e\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") "
Oct 30 04:23:10.407259 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.406951    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-envoy\") pod \"9f813b35-b180-40dd-bdd1-91fffa7e133e\" (UID: \"9f813b35-b180-40dd-bdd1-91fffa7e133e\") "
Oct 30 04:23:10.407472 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407429    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.407472 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407455    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.407472 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407468    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.407648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407481    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f813b35-b180-40dd-bdd1-91fffa7e133e-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.407648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407494    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.407648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407506    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.407648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407519    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.407648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407532    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.407648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407546    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.407648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407559    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfcbc\" (UniqueName: \"kubernetes.io/projected/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-kube-api-access-tfcbc\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.407648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407572    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.407648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407584    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.407648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.407597    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f88e7d-21c3-4e59-9665-5cb24a87b5d5-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.408594 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.408568    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-token" pod "9f813b35-b180-40dd-bdd1-91fffa7e133e" (UID: "9f813b35-b180-40dd-bdd1-91fffa7e133e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:23:10.408842 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.408820    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-kube-api-access-lzd4z" pod "9f813b35-b180-40dd-bdd1-91fffa7e133e" (UID: "9f813b35-b180-40dd-bdd1-91fffa7e133e"). InnerVolumeSpecName "kube-api-access-lzd4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:23:10.409002 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.408985    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-podinfo" pod "9f813b35-b180-40dd-bdd1-91fffa7e133e" (UID: "9f813b35-b180-40dd-bdd1-91fffa7e133e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:23:10.409361 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.409347    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-envoy" pod "9f813b35-b180-40dd-bdd1-91fffa7e133e" (UID: "9f813b35-b180-40dd-bdd1-91fffa7e133e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:10.508888 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.508763    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.508888 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.508797    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.508888 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.508814    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzd4z\" (UniqueName: \"kubernetes.io/projected/9f813b35-b180-40dd-bdd1-91fffa7e133e-kube-api-access-lzd4z\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:10.508888 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:10.508830    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f813b35-b180-40dd-bdd1-91fffa7e133e-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:23:11.244550 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:11.244456    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"
Oct 30 04:23:11.244982 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:11.244455    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"
Oct 30 04:23:11.312320 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:11.312289    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"]
Oct 30 04:23:11.324867 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:11.324837    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-frvgg"]
Oct 30 04:23:11.350661 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:11.350632    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"]
Oct 30 04:23:11.357271 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:11.357249    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flqhx"]
Oct 30 04:23:11.752323 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:23:11.752291    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:23:12.408335 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:12.408297    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f88e7d-21c3-4e59-9665-5cb24a87b5d5" path="/var/lib/kubelet/pods/27f88e7d-21c3-4e59-9665-5cb24a87b5d5/volumes"
Oct 30 04:23:12.408815 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:12.408801    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f813b35-b180-40dd-bdd1-91fffa7e133e" path="/var/lib/kubelet/pods/9f813b35-b180-40dd-bdd1-91fffa7e133e/volumes"
Oct 30 04:23:31.683385 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.683350    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"]
Oct 30 04:23:31.683904 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.683889    5072 cpu_manager.go:414] "containerMap: removing container" podUID="9f813b35-b180-40dd-bdd1-91fffa7e133e" containerName="istio-proxy"
Oct 30 04:23:31.683961 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.683905    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f813b35-b180-40dd-bdd1-91fffa7e133e" containerName="istio-proxy"
Oct 30 04:23:31.683961 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.683954    5072 cpu_manager.go:414] "containerMap: removing container" podUID="27f88e7d-21c3-4e59-9665-5cb24a87b5d5" containerName="istio-proxy"
Oct 30 04:23:31.683961 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.683958    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f88e7d-21c3-4e59-9665-5cb24a87b5d5" containerName="istio-proxy"
Oct 30 04:23:31.684067 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.684035    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f88e7d-21c3-4e59-9665-5cb24a87b5d5" containerName="istio-proxy"
Oct 30 04:23:31.684067 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.684044    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f813b35-b180-40dd-bdd1-91fffa7e133e" containerName="istio-proxy"
Oct 30 04:23:31.687726 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.687709    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:31.706477 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.706448    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"]
Oct 30 04:23:31.718424 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.718384    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1a0d3427-serving-cert\""
Oct 30 04:23:31.774156 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.774123    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btcn\" (UniqueName: \"kubernetes.io/projected/94072976-07f9-412d-bf5c-674d66ad42c8-kube-api-access-5btcn\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:31.774331 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.774281    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94072976-07f9-412d-bf5c-674d66ad42c8-openshift-service-ca\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:31.774399 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.774382    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-maas-api-tls\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:31.774439 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.774427    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-metrics-certs\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:31.875984 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.875954    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-maas-api-tls\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:31.875984 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.875991    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-metrics-certs\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:31.876261 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.876041    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btcn\" (UniqueName: \"kubernetes.io/projected/94072976-07f9-412d-bf5c-674d66ad42c8-kube-api-access-5btcn\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:31.876261 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.876156    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94072976-07f9-412d-bf5c-674d66ad42c8-openshift-service-ca\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:31.876806 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.876746    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94072976-07f9-412d-bf5c-674d66ad42c8-openshift-service-ca\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:31.878657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.878637    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-maas-api-tls\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:31.878747 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.878689    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-metrics-certs\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:31.900510 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:31.900476    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btcn\" (UniqueName: \"kubernetes.io/projected/94072976-07f9-412d-bf5c-674d66ad42c8-kube-api-access-5btcn\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:32.001219 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:32.001182    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:32.307217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:32.307182    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"]
Oct 30 04:23:32.307914 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:23:32.307887    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94072976_07f9_412d_bf5c_674d66ad42c8.slice/crio-9e924552e61af3946c6bc6789ac48cb5b9867b7b9bfef95e9cb23687041d8f75 WatchSource:0}: Error finding container 9e924552e61af3946c6bc6789ac48cb5b9867b7b9bfef95e9cb23687041d8f75: Status 404 returned error can't find the container with id 9e924552e61af3946c6bc6789ac48cb5b9867b7b9bfef95e9cb23687041d8f75
Oct 30 04:23:32.373971 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:32.373942    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w" event={"ID":"94072976-07f9-412d-bf5c-674d66ad42c8","Type":"ContainerStarted","Data":"9e924552e61af3946c6bc6789ac48cb5b9867b7b9bfef95e9cb23687041d8f75"}
Oct 30 04:23:33.382805 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:33.382769    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w" event={"ID":"94072976-07f9-412d-bf5c-674d66ad42c8","Type":"ContainerStarted","Data":"0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639"}
Oct 30 04:23:33.383243 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:33.382893    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:33.427097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:33.427038    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w" podStartSLOduration=1.9494737880000002 podStartE2EDuration="2.427024919s" totalImagesPullingTime="477.551131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:23:33.426245034 +0000 UTC m=+1593.167668157" watchObservedRunningTime="2026-10-30 04:23:33.427024919 +0000 UTC m=+1593.168448016"
Oct 30 04:23:39.395213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:39.395185    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:23:41.773874 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:23:41.773831    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:23:51.038886 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.038847    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"]
Oct 30 04:23:51.043997 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.043972    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.060542 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.060514    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"]
Oct 30 04:23:51.070415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.070386    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1a0d3427\"/\"openshift-service-ca.crt\""
Oct 30 04:23:51.070617 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.070439    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1a0d3427\"/\"kube-root-ca.crt\""
Oct 30 04:23:51.071954 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.071937    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1a0d3427\"/\"default-dockercfg-bt9ww\""
Oct 30 04:23:51.072035 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.071957    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1a0d3427\"/\"e2e-lifecycle-model-1a0d3427-kserve-self-signed-certs\""
Oct 30 04:23:51.112435 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.112399    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-model-cache\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.112435 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.112444    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-dshm\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.112722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.112588    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdhgm\" (UniqueName: \"kubernetes.io/projected/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-kube-api-access-hdhgm\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.112722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.112697    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-home\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.112722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.112722    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tmp-dir\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.112872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.112769    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tls-certs\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.213455 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.213417    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-model-cache\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.213455 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.213461    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-dshm\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.213724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.213524    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdhgm\" (UniqueName: \"kubernetes.io/projected/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-kube-api-access-hdhgm\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.213724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.213596    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-home\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.213724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.213615    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tmp-dir\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.213724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.213665    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tls-certs\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.213983 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.213961    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-model-cache\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.214048 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.214013    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tmp-dir\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.214128 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.214107    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-home\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.216352 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.216327    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-dshm\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.216674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.216653    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tls-certs\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.238023 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.237991    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdhgm\" (UniqueName: \"kubernetes.io/projected/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-kube-api-access-hdhgm\") pod \"e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") " pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.363042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.362940    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:23:51.671278 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:51.671247    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"]
Oct 30 04:23:51.672102 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:23:51.672060    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef61e149_c7e9_4ea8_ab94_82cd19fdd4b1.slice/crio-d4cd7238ed7c8e728eb0eb1d2f588afa9613d7674dd0ceea7309832ffeae4974 WatchSource:0}: Error finding container d4cd7238ed7c8e728eb0eb1d2f588afa9613d7674dd0ceea7309832ffeae4974: Status 404 returned error can't find the container with id d4cd7238ed7c8e728eb0eb1d2f588afa9613d7674dd0ceea7309832ffeae4974
Oct 30 04:23:52.499742 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:52.499704    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt" event={"ID":"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1","Type":"ContainerStarted","Data":"a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb"}
Oct 30 04:23:52.500302 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:52.499754    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt" event={"ID":"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1","Type":"ContainerStarted","Data":"d4cd7238ed7c8e728eb0eb1d2f588afa9613d7674dd0ceea7309832ffeae4974"}
Oct 30 04:23:52.544769 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:52.544715    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt" podStartSLOduration=1.544699201 podStartE2EDuration="1.544699201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:23:52.541563006 +0000 UTC m=+1612.282986106" watchObservedRunningTime="2026-10-30 04:23:52.544699201 +0000 UTC m=+1612.286122356"
Oct 30 04:23:53.363178 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:23:53.363141    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:24:00.646619 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:00.646587    5072 scope.go:122] "RemoveContainer" containerID="ff6d95cfc32516967d9e1396ced5a116b3a0a7f9601d0268e0fa72b9b77aaa4a"
Oct 30 04:24:00.658062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:00.658039    5072 scope.go:122] "RemoveContainer" containerID="d810ba66e358f5b7a5e1259a2b5e9bfa89349bb45b8b97bb8bd2ea4765d181da"
Oct 30 04:24:01.363246 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:01.363213    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:24:01.376441 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:01.376412    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:24:01.566902 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:01.566870    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:24:05.206533 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.206501    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"]
Oct 30 04:24:05.212202 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.212177    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.240799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.240769    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"]
Oct 30 04:24:05.391069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.391032    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.391284 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.391104    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090baa36-2795-4057-9963-5f9a0e448b55-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.391284 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.391203    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.391284 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.391278    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjb9p\" (UniqueName: \"kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-kube-api-access-xjb9p\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.391466 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.391446    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.391541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.391525    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.391598 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.391569    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090baa36-2795-4057-9963-5f9a0e448b55-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.391669 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.391650    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.391757 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.391730    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.493315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.493232    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.493478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.493320    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.493478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.493346    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090baa36-2795-4057-9963-5f9a0e448b55-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.493478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.493379    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.493478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.493453    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.493692 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.493512    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.493692 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.493552    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090baa36-2795-4057-9963-5f9a0e448b55-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.493692 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.493582    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.493692 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.493638    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjb9p\" (UniqueName: \"kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-kube-api-access-xjb9p\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.493965 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.493729    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.494143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.494062    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.494143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.494122    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.494375 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.494360    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090baa36-2795-4057-9963-5f9a0e448b55-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.494447 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.494428    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.495963 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.495941    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.496495 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.496478    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090baa36-2795-4057-9963-5f9a0e448b55-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.518597 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.518567    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.518712 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.518608    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjb9p\" (UniqueName: \"kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-kube-api-access-xjb9p\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sw4ms\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.530698 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.530677    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:05.853352 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.853330    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"]
Oct 30 04:24:05.853944 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:24:05.853909    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090baa36_2795_4057_9963_5f9a0e448b55.slice/crio-c2cb1e5fd05f053cfb15929c8e54bfca5b746b97d95ac542d1151a6f8e33f272 WatchSource:0}: Error finding container c2cb1e5fd05f053cfb15929c8e54bfca5b746b97d95ac542d1151a6f8e33f272: Status 404 returned error can't find the container with id c2cb1e5fd05f053cfb15929c8e54bfca5b746b97d95ac542d1151a6f8e33f272
Oct 30 04:24:05.856296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.856266    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:24:05.856383 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.856338    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:24:05.856383 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:05.856367    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:24:06.593768 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:06.593735    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" event={"ID":"090baa36-2795-4057-9963-5f9a0e448b55","Type":"ContainerStarted","Data":"ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca"}
Oct 30 04:24:06.594204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:06.593800    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" event={"ID":"090baa36-2795-4057-9963-5f9a0e448b55","Type":"ContainerStarted","Data":"c2cb1e5fd05f053cfb15929c8e54bfca5b746b97d95ac542d1151a6f8e33f272"}
Oct 30 04:24:06.653047 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:06.652998    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" podStartSLOduration=1.652985092 podStartE2EDuration="1.652985092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:06.652034016 +0000 UTC m=+1626.393457115" watchObservedRunningTime="2026-10-30 04:24:06.652985092 +0000 UTC m=+1626.394408189"
Oct 30 04:24:07.531773 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:07.531738    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:07.535212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:07.535184    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sw4ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 04:24:07.535514 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:07.535490    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" podUID="090baa36-2795-4057-9963-5f9a0e448b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 04:24:08.531830 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:08.531789    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sw4ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 04:24:08.532361 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:08.531856    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" podUID="090baa36-2795-4057-9963-5f9a0e448b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 04:24:09.531488 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:09.531455    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sw4ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 04:24:09.531675 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:09.531516    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" podUID="090baa36-2795-4057-9963-5f9a0e448b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 04:24:10.531320 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:10.531276    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sw4ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 04:24:10.531801 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:10.531338    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" podUID="090baa36-2795-4057-9963-5f9a0e448b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 04:24:11.531435 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:11.531407    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sw4ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 04:24:11.531833 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:11.531453    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" podUID="090baa36-2795-4057-9963-5f9a0e448b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 04:24:11.751451 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:24:11.751418    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:24:12.531544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:12.531503    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sw4ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 04:24:12.532000 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:12.531577    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" podUID="090baa36-2795-4057-9963-5f9a0e448b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 04:24:13.531605 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:13.531577    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sw4ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 04:24:13.532001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:13.531638    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" podUID="090baa36-2795-4057-9963-5f9a0e448b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 04:24:14.596434 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:14.596392    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:14.598427 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:14.598404    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:14.601510 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:14.601484    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:24:20.265615 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.265582    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"]
Oct 30 04:24:20.270143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.270115    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.294388 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.294363    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"]
Oct 30 04:24:20.306774 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.306750    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkgmx\" (UniqueName: \"kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-kube-api-access-xkgmx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.306910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.306787    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.306910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.306885    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.307029 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.306937    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.307105 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.307049    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.307170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.307137    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.307225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.307204    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.307282 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.307246    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.307525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.307323    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.408411 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.408383    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkgmx\" (UniqueName: \"kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-kube-api-access-xkgmx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.408582 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.408419    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.408582 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.408547    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.408694 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.408603    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.408694 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.408659    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.408797 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.408700    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.408797 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.408764    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.408909 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.408806    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.408909 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.408827    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.408909 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.408844    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.409067 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.409012    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.409158 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.409104    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.409271 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.409249    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.409630 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.409596    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.411437 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.411415    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.411437 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.411425    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.446783 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.446757    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.446884 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.446807    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkgmx\" (UniqueName: \"kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-kube-api-access-xkgmx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-987vr\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.586327 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.586259    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:20.907644 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:24:20.907614    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8168654_1ce2_4b2b_b09a_42c08d07b31d.slice/crio-52f8a4c6761facc51edc3a1c1e7c055293b55ffe5159014d0dabc602c9205f7d WatchSource:0}: Error finding container 52f8a4c6761facc51edc3a1c1e7c055293b55ffe5159014d0dabc602c9205f7d: Status 404 returned error can't find the container with id 52f8a4c6761facc51edc3a1c1e7c055293b55ffe5159014d0dabc602c9205f7d
Oct 30 04:24:20.908030 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.908009    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"]
Oct 30 04:24:20.910224 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.910193    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:24:20.910300 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.910259    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:24:20.910300 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:20.910292    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:24:21.699735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:21.699688    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" event={"ID":"b8168654-1ce2-4b2b-b09a-42c08d07b31d","Type":"ContainerStarted","Data":"2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498"}
Oct 30 04:24:21.700288 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:21.699802    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" event={"ID":"b8168654-1ce2-4b2b-b09a-42c08d07b31d","Type":"ContainerStarted","Data":"52f8a4c6761facc51edc3a1c1e7c055293b55ffe5159014d0dabc602c9205f7d"}
Oct 30 04:24:21.783738 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:21.783675    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" podStartSLOduration=1.7836563600000002 podStartE2EDuration="1.78365636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:21.780705398 +0000 UTC m=+1641.522128495" watchObservedRunningTime="2026-10-30 04:24:21.78365636 +0000 UTC m=+1641.525079458"
Oct 30 04:24:22.586733 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:22.586695    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:22.588448 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:22.588425    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-987vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 04:24:22.588539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:22.588478    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" podUID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 04:24:23.586912 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:23.586875    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-987vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 04:24:23.587328 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:23.586939    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" podUID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 04:24:24.589128 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:24.587925    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-987vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 04:24:24.589128 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:24.588022    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" podUID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 04:24:25.586581 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:25.586545    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-987vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 04:24:25.586759 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:25.586604    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" podUID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 04:24:26.195140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.195105    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"]
Oct 30 04:24:26.200520 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.200493    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.222756 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.222731    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"]
Oct 30 04:24:26.230136 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.230116    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e3c43b-serving-cert\""
Oct 30 04:24:26.302124 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.302059    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-metrics-certs\") pod \"maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.302309 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.302260    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/730f9801-4975-413e-98c9-e4c5b83a6c74-openshift-service-ca\") pod \"maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.302309 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.302297    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-maas-api-tls\") pod \"maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.302417 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.302331    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h947t\" (UniqueName: \"kubernetes.io/projected/730f9801-4975-413e-98c9-e4c5b83a6c74-kube-api-access-h947t\") pod \"maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.404973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.404928    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/730f9801-4975-413e-98c9-e4c5b83a6c74-openshift-service-ca\") pod \"maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.405213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.405013    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-maas-api-tls\") pod \"maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.405213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.405095    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h947t\" (UniqueName: \"kubernetes.io/projected/730f9801-4975-413e-98c9-e4c5b83a6c74-kube-api-access-h947t\") pod \"maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.405213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.405187    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-metrics-certs\") pod \"maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.406127 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.406059    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/730f9801-4975-413e-98c9-e4c5b83a6c74-openshift-service-ca\") pod \"maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.409660 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.409633    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-metrics-certs\") pod \"maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.409771 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.409673    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-maas-api-tls\") pod \"maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.430645 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.430621    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h947t\" (UniqueName: \"kubernetes.io/projected/730f9801-4975-413e-98c9-e4c5b83a6c74-kube-api-access-h947t\") pod \"maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.514727 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.514640    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:26.587667 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.587624    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-987vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 04:24:26.587846 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.587688    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" podUID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 04:24:26.848639 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.848611    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"]
Oct 30 04:24:26.850016 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:26.849997    5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:24:27.587441 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:27.587406    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-987vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 04:24:27.587827 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:27.587466    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" podUID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 04:24:27.738975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:27.738937    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm" event={"ID":"730f9801-4975-413e-98c9-e4c5b83a6c74","Type":"ContainerStarted","Data":"3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256"}
Oct 30 04:24:27.739196 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:27.738993    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm" event={"ID":"730f9801-4975-413e-98c9-e4c5b83a6c74","Type":"ContainerStarted","Data":"128c8666106fe42d1c453fb21ada7c3612c7cb0dac6df150af4ba9bb452689f0"}
Oct 30 04:24:27.739196 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:27.739024    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:27.786462 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:27.786396    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm" podStartSLOduration=1.459047286 podStartE2EDuration="1.786377165s" totalImagesPullingTime="327.329879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:27.78221848 +0000 UTC m=+1647.523641578" watchObservedRunningTime="2026-10-30 04:24:27.786377165 +0000 UTC m=+1647.527800262"
Oct 30 04:24:28.587584 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:28.587551    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-987vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 04:24:28.587988 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:28.587606    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" podUID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 04:24:29.637516 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:29.637489    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:29.637909 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:29.637814    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:29.638829 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:29.638796    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:24:31.028251 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:31.028221    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"]
Oct 30 04:24:31.040656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:31.040630    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ba317c-sj5q9"]
Oct 30 04:24:32.408738 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:32.408684    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d14cae-d09f-4340-9ede-2aa115b7d59d" path="/var/lib/kubelet/pods/16d14cae-d09f-4340-9ede-2aa115b7d59d/volumes"
Oct 30 04:24:33.750022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:33.749989    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:24:41.751962 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:24:41.751878    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:24:51.643240 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:51.643197    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11264-cd7602"]
Oct 30 04:24:51.647318 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:51.647298    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11264-cd7602"
Oct 30 04:24:51.668381 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:51.668355    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gsgvm\""
Oct 30 04:24:51.672204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:51.672181    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11264-cd7602"]
Oct 30 04:24:51.721614 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:51.721585    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr2sd\" (UniqueName: \"kubernetes.io/projected/f471140c-0068-4dc3-b118-e99faf32ca30-kube-api-access-wr2sd\") pod \"test-curl-11264-cd7602\" (UID: \"f471140c-0068-4dc3-b118-e99faf32ca30\") " pod="opendatahub/test-curl-11264-cd7602"
Oct 30 04:24:51.822807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:51.822765    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr2sd\" (UniqueName: \"kubernetes.io/projected/f471140c-0068-4dc3-b118-e99faf32ca30-kube-api-access-wr2sd\") pod \"test-curl-11264-cd7602\" (UID: \"f471140c-0068-4dc3-b118-e99faf32ca30\") " pod="opendatahub/test-curl-11264-cd7602"
Oct 30 04:24:51.847512 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:51.847483    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr2sd\" (UniqueName: \"kubernetes.io/projected/f471140c-0068-4dc3-b118-e99faf32ca30-kube-api-access-wr2sd\") pod \"test-curl-11264-cd7602\" (UID: \"f471140c-0068-4dc3-b118-e99faf32ca30\") " pod="opendatahub/test-curl-11264-cd7602"
Oct 30 04:24:51.959953 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:51.959928    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11264-cd7602"
Oct 30 04:24:52.260058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:52.260029    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11264-cd7602"]
Oct 30 04:24:52.260774 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:24:52.260741    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf471140c_0068_4dc3_b118_e99faf32ca30.slice/crio-4e9c4dae17bf3284a7ee3d29c6c8c51a06369f7d13646ad7d69bce58b4775016 WatchSource:0}: Error finding container 4e9c4dae17bf3284a7ee3d29c6c8c51a06369f7d13646ad7d69bce58b4775016: Status 404 returned error can't find the container with id 4e9c4dae17bf3284a7ee3d29c6c8c51a06369f7d13646ad7d69bce58b4775016
Oct 30 04:24:52.916744 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:52.916699    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11264-cd7602" event={"ID":"f471140c-0068-4dc3-b118-e99faf32ca30","Type":"ContainerStarted","Data":"4e9c4dae17bf3284a7ee3d29c6c8c51a06369f7d13646ad7d69bce58b4775016"}
Oct 30 04:24:54.931960 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:54.931916    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11264-cd7602" event={"ID":"f471140c-0068-4dc3-b118-e99faf32ca30","Type":"ContainerStarted","Data":"3eb05f98802c1c9a554b939a11e48c8078e67455067cfc205d75626159c5ec22"}
Oct 30 04:24:54.974289 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:54.974208    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11264-cd7602" podStartSLOduration=1.853647546 podStartE2EDuration="3.974186771s" totalImagesPullingTime="2.120539225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:54.971770682 +0000 UTC m=+1674.713193779" watchObservedRunningTime="2026-10-30 04:24:54.974186771 +0000 UTC m=+1674.715609872"
Oct 30 04:24:55.863925 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:55.863892    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11264-cd7602"]
Oct 30 04:24:55.868124 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:55.867353    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11264-cd7602"]
Oct 30 04:24:56.944759 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:56.944720    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11264-cd7602" podUID="f471140c-0068-4dc3-b118-e99faf32ca30" containerName="test-curl-11264-cd7602" containerID="cri-o://3eb05f98802c1c9a554b939a11e48c8078e67455067cfc205d75626159c5ec22" gracePeriod=2
Oct 30 04:24:59.159309 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:59.159275    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11264-cd7602"
Oct 30 04:24:59.273122 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:59.273024    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f471140c-0068-4dc3-b118-e99faf32ca30-kube-api-access-wr2sd\" (UniqueName: \"kubernetes.io/projected/f471140c-0068-4dc3-b118-e99faf32ca30-kube-api-access-wr2sd\") pod \"f471140c-0068-4dc3-b118-e99faf32ca30\" (UID: \"f471140c-0068-4dc3-b118-e99faf32ca30\") "
Oct 30 04:24:59.275270 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:59.275242    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f471140c-0068-4dc3-b118-e99faf32ca30-kube-api-access-wr2sd" pod "f471140c-0068-4dc3-b118-e99faf32ca30" (UID: "f471140c-0068-4dc3-b118-e99faf32ca30"). InnerVolumeSpecName "kube-api-access-wr2sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:24:59.374876 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:59.374841    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr2sd\" (UniqueName: \"kubernetes.io/projected/f471140c-0068-4dc3-b118-e99faf32ca30-kube-api-access-wr2sd\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:24:59.963492 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:59.963457    5072 generic.go:356] "Generic (PLEG): container finished" podID="f471140c-0068-4dc3-b118-e99faf32ca30" containerID="3eb05f98802c1c9a554b939a11e48c8078e67455067cfc205d75626159c5ec22" exitCode=137
Oct 30 04:24:59.963682 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:59.963508    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11264-cd7602"
Oct 30 04:24:59.963682 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:59.963517    5072 scope.go:122] "RemoveContainer" containerID="3eb05f98802c1c9a554b939a11e48c8078e67455067cfc205d75626159c5ec22"
Oct 30 04:24:59.973517 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:59.973499    5072 scope.go:122] "RemoveContainer" containerID="3eb05f98802c1c9a554b939a11e48c8078e67455067cfc205d75626159c5ec22"
Oct 30 04:24:59.973795 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:24:59.973772    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb05f98802c1c9a554b939a11e48c8078e67455067cfc205d75626159c5ec22\": container with ID starting with 3eb05f98802c1c9a554b939a11e48c8078e67455067cfc205d75626159c5ec22 not found: ID does not exist" containerID="3eb05f98802c1c9a554b939a11e48c8078e67455067cfc205d75626159c5ec22"
Oct 30 04:24:59.973913 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:24:59.973801    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb05f98802c1c9a554b939a11e48c8078e67455067cfc205d75626159c5ec22"} err="failed to get container status \"3eb05f98802c1c9a554b939a11e48c8078e67455067cfc205d75626159c5ec22\": rpc error: code = NotFound desc = could not find container \"3eb05f98802c1c9a554b939a11e48c8078e67455067cfc205d75626159c5ec22\": container with ID starting with 3eb05f98802c1c9a554b939a11e48c8078e67455067cfc205d75626159c5ec22 not found: ID does not exist"
Oct 30 04:24:59.986041 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:24:59.986019    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11264-cd7602\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="f471140c-0068-4dc3-b118-e99faf32ca30" pod="opendatahub/test-curl-11264-cd7602"
Oct 30 04:25:00.409059 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:00.408987    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f471140c-0068-4dc3-b118-e99faf32ca30" path="/var/lib/kubelet/pods/f471140c-0068-4dc3-b118-e99faf32ca30/volumes"
Oct 30 04:25:00.420955 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:25:00.420925    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11264-cd7602\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="f471140c-0068-4dc3-b118-e99faf32ca30" pod="opendatahub/test-curl-11264-cd7602"
Oct 30 04:25:09.663442 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.663395    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"]
Oct 30 04:25:09.664240 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.664213    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f471140c-0068-4dc3-b118-e99faf32ca30" containerName="test-curl-11264-cd7602"
Oct 30 04:25:09.664240 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.664240    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f471140c-0068-4dc3-b118-e99faf32ca30" containerName="test-curl-11264-cd7602"
Oct 30 04:25:09.664555 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.664438    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f471140c-0068-4dc3-b118-e99faf32ca30" containerName="test-curl-11264-cd7602"
Oct 30 04:25:09.669626 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.669606    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.687091 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.687048    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"]
Oct 30 04:25:09.695496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.695453    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-397bda-serving-cert\""
Oct 30 04:25:09.794871 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.794837    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw768\" (UniqueName: \"kubernetes.io/projected/4dd61d11-85e2-499d-af65-dfd7fdedcf53-kube-api-access-cw768\") pod \"maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.795050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.794877    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-metrics-certs\") pod \"maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.795050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.794953    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd61d11-85e2-499d-af65-dfd7fdedcf53-openshift-service-ca\") pod \"maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.795050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.794999    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-maas-api-tls\") pod \"maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.896031 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.895996    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw768\" (UniqueName: \"kubernetes.io/projected/4dd61d11-85e2-499d-af65-dfd7fdedcf53-kube-api-access-cw768\") pod \"maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.896238 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.896040    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-metrics-certs\") pod \"maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.896238 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.896134    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd61d11-85e2-499d-af65-dfd7fdedcf53-openshift-service-ca\") pod \"maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.896238 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.896194    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-maas-api-tls\") pod \"maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.896832 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.896808    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd61d11-85e2-499d-af65-dfd7fdedcf53-openshift-service-ca\") pod \"maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.898729 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.898703    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-metrics-certs\") pod \"maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.898822 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.898731    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-maas-api-tls\") pod \"maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.923482 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.923407    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw768\" (UniqueName: \"kubernetes.io/projected/4dd61d11-85e2-499d-af65-dfd7fdedcf53-kube-api-access-cw768\") pod \"maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:09.984562 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:09.984507    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:10.290203 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:25:10.290162    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd61d11_85e2_499d_af65_dfd7fdedcf53.slice/crio-8545ea98836454fa3650da05c99da324ac99a92a39f5f6d3dc0a64fe8b0ffe05 WatchSource:0}: Error finding container 8545ea98836454fa3650da05c99da324ac99a92a39f5f6d3dc0a64fe8b0ffe05: Status 404 returned error can't find the container with id 8545ea98836454fa3650da05c99da324ac99a92a39f5f6d3dc0a64fe8b0ffe05
Oct 30 04:25:10.291030 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:10.291001    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"]
Oct 30 04:25:11.031650 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:11.031603    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9" event={"ID":"4dd61d11-85e2-499d-af65-dfd7fdedcf53","Type":"ContainerStarted","Data":"8c333b35bf7d49a10dcebf7ac5406fcd53d02ed6828511424ebb6c23ee3995d5"}
Oct 30 04:25:11.032161 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:11.031674    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9" event={"ID":"4dd61d11-85e2-499d-af65-dfd7fdedcf53","Type":"ContainerStarted","Data":"8545ea98836454fa3650da05c99da324ac99a92a39f5f6d3dc0a64fe8b0ffe05"}
Oct 30 04:25:11.032161 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:11.031856    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:11.073678 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:11.073636    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9" podStartSLOduration=1.728876889 podStartE2EDuration="2.073622248s" totalImagesPullingTime="344.745359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:25:11.07278519 +0000 UTC m=+1690.814208288" watchObservedRunningTime="2026-10-30 04:25:11.073622248 +0000 UTC m=+1690.815045346"
Oct 30 04:25:11.751655 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:25:11.751621    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:25:15.017901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.017862    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"]
Oct 30 04:25:15.022507 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.022481    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.044715 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.044690    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"]
Oct 30 04:25:15.053909 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.053885    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 04:25:15.054110 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.053899    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 04:25:15.054110 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.053905    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 04:25:15.054247 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.053923    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2vrdc\""
Oct 30 04:25:15.159604 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.159578    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.159764 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.159612    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvt5\" (UniqueName: \"kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-kube-api-access-5cvt5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.159764 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.159727    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.159857 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.159770    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.159857 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.159809    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.159924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.159895    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.159960 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.159939    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.261128 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.261098    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.261292 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.261135    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvt5\" (UniqueName: \"kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-kube-api-access-5cvt5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.261292 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.261191    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.261292 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.261219    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.261491 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.261286    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.261491 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.261354    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.261491 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.261403    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.261977 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.261952    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.263620 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.263595    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.263791 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.263770    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.263867 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.263836    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.263915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.263874    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.284782 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.284725    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.284969 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.284949    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvt5\" (UniqueName: \"kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-kube-api-access-5cvt5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j24vr\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.336683 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.336659    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:15.652872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:15.652847    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"]
Oct 30 04:25:15.653131 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:25:15.653109    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9172cb2_60dd_4c80_8704_b5e7e82e5fdc.slice/crio-b2b3a7efa3edd2aeb877a4654d775aff10049931ae0256ea8dfc641a15ae9257 WatchSource:0}: Error finding container b2b3a7efa3edd2aeb877a4654d775aff10049931ae0256ea8dfc641a15ae9257: Status 404 returned error can't find the container with id b2b3a7efa3edd2aeb877a4654d775aff10049931ae0256ea8dfc641a15ae9257
Oct 30 04:25:16.067599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:16.067564    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr" event={"ID":"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc","Type":"ContainerStarted","Data":"b2b3a7efa3edd2aeb877a4654d775aff10049931ae0256ea8dfc641a15ae9257"}
Oct 30 04:25:17.045059 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:17.045029    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:25:18.908596 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:18.908566    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:25:18.908885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:18.908628    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:25:19.100357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:19.100322    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr" event={"ID":"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc","Type":"ContainerStarted","Data":"1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3"}
Oct 30 04:25:19.100565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:19.100482    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:19.102304 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:19.102261    5072 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-j24vr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:25:19.102480 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:19.102311    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr" podUID="e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:25:19.161140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:19.161022    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr" podStartSLOduration=1.908701954 podStartE2EDuration="5.161006149s" totalImagesPullingTime="3.252304195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:25:19.157869628 +0000 UTC m=+1698.899292727" watchObservedRunningTime="2026-10-30 04:25:19.161006149 +0000 UTC m=+1698.902429246"
Oct 30 04:25:20.107556 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:20.107531    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:25:41.816314 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:25:41.816271    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:25:43.917804 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:43.917774    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"]
Oct 30 04:25:43.918287 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:43.918013    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w" podUID="94072976-07f9-412d-bf5c-674d66ad42c8" containerName="maas-api" containerID="cri-o://0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639" gracePeriod=30
Oct 30 04:25:44.237598 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.237561    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:25:44.275302 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.275273    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-metrics-certs\") pod \"94072976-07f9-412d-bf5c-674d66ad42c8\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") "
Oct 30 04:25:44.275493 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.275404    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94072976-07f9-412d-bf5c-674d66ad42c8-kube-api-access-5btcn\" (UniqueName: \"kubernetes.io/projected/94072976-07f9-412d-bf5c-674d66ad42c8-kube-api-access-5btcn\") pod \"94072976-07f9-412d-bf5c-674d66ad42c8\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") "
Oct 30 04:25:44.275493 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.275477    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94072976-07f9-412d-bf5c-674d66ad42c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94072976-07f9-412d-bf5c-674d66ad42c8-openshift-service-ca\") pod \"94072976-07f9-412d-bf5c-674d66ad42c8\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") "
Oct 30 04:25:44.275606 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.275508    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-maas-api-tls\") pod \"94072976-07f9-412d-bf5c-674d66ad42c8\" (UID: \"94072976-07f9-412d-bf5c-674d66ad42c8\") "
Oct 30 04:25:44.275936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.275860    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94072976-07f9-412d-bf5c-674d66ad42c8-openshift-service-ca" pod "94072976-07f9-412d-bf5c-674d66ad42c8" (UID: "94072976-07f9-412d-bf5c-674d66ad42c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:25:44.276211 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.276065    5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94072976-07f9-412d-bf5c-674d66ad42c8-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:44.277627 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.277603    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-metrics-certs" pod "94072976-07f9-412d-bf5c-674d66ad42c8" (UID: "94072976-07f9-412d-bf5c-674d66ad42c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:25:44.277844 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.277816    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-maas-api-tls" pod "94072976-07f9-412d-bf5c-674d66ad42c8" (UID: "94072976-07f9-412d-bf5c-674d66ad42c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:25:44.277844 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.277838    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94072976-07f9-412d-bf5c-674d66ad42c8-kube-api-access-5btcn" pod "94072976-07f9-412d-bf5c-674d66ad42c8" (UID: "94072976-07f9-412d-bf5c-674d66ad42c8"). InnerVolumeSpecName "kube-api-access-5btcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:44.324799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.324050    5072 generic.go:356] "Generic (PLEG): container finished" podID="94072976-07f9-412d-bf5c-674d66ad42c8" containerID="0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639" exitCode=0
Oct 30 04:25:44.324799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.324230    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w" event={"ID":"94072976-07f9-412d-bf5c-674d66ad42c8","Type":"ContainerDied","Data":"0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639"}
Oct 30 04:25:44.324799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.324334    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w" event={"ID":"94072976-07f9-412d-bf5c-674d66ad42c8","Type":"ContainerDied","Data":"9e924552e61af3946c6bc6789ac48cb5b9867b7b9bfef95e9cb23687041d8f75"}
Oct 30 04:25:44.324799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.324382    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"
Oct 30 04:25:44.324799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.324402    5072 scope.go:122] "RemoveContainer" containerID="0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639"
Oct 30 04:25:44.340018 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.339995    5072 scope.go:122] "RemoveContainer" containerID="0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639"
Oct 30 04:25:44.340632 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:25:44.340606    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639\": container with ID starting with 0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639 not found: ID does not exist" containerID="0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639"
Oct 30 04:25:44.340738 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.340646    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639"} err="failed to get container status \"0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639\": rpc error: code = NotFound desc = could not find container \"0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639\": container with ID starting with 0eeabe9eac879dd8c28b2b9ac0378a2e4695d7b6a8b8dc3e4410d88da63d9639 not found: ID does not exist"
Oct 30 04:25:44.377372 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.377343    5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-metrics-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:44.377496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.377375    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5btcn\" (UniqueName: \"kubernetes.io/projected/94072976-07f9-412d-bf5c-674d66ad42c8-kube-api-access-5btcn\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:44.377496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.377390    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94072976-07f9-412d-bf5c-674d66ad42c8-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:44.385508 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.385484    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"]
Oct 30 04:25:44.389645 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.389611    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-tmw7w"]
Oct 30 04:25:44.409477 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:44.409446    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94072976-07f9-412d-bf5c-674d66ad42c8" path="/var/lib/kubelet/pods/94072976-07f9-412d-bf5c-674d66ad42c8/volumes"
Oct 30 04:25:51.635332 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:51.635302    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"]
Oct 30 04:25:51.635761 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:51.635497    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7" podUID="1e5e829f-493e-46f1-84de-c77c944d1108" containerName="istio-proxy" containerID="cri-o://7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614" gracePeriod=30
Oct 30 04:25:51.953543 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:51.953507    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"]
Oct 30 04:25:51.953786 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:51.953752    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm" podUID="730f9801-4975-413e-98c9-e4c5b83a6c74" containerName="maas-api" containerID="cri-o://3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256" gracePeriod=30
Oct 30 04:25:52.332527 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.332486    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:25:52.380160 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.379965    5072 generic.go:356] "Generic (PLEG): container finished" podID="730f9801-4975-413e-98c9-e4c5b83a6c74" containerID="3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256" exitCode=0
Oct 30 04:25:52.380160 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.380027    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"
Oct 30 04:25:52.380160 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.380051    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm" event={"ID":"730f9801-4975-413e-98c9-e4c5b83a6c74","Type":"ContainerDied","Data":"3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256"}
Oct 30 04:25:52.380478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.380172    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm" event={"ID":"730f9801-4975-413e-98c9-e4c5b83a6c74","Type":"ContainerDied","Data":"128c8666106fe42d1c453fb21ada7c3612c7cb0dac6df150af4ba9bb452689f0"}
Oct 30 04:25:52.380478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.380206    5072 scope.go:122] "RemoveContainer" containerID="3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256"
Oct 30 04:25:52.385712 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.385656    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730f9801-4975-413e-98c9-e4c5b83a6c74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/730f9801-4975-413e-98c9-e4c5b83a6c74-openshift-service-ca\") pod \"730f9801-4975-413e-98c9-e4c5b83a6c74\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") "
Oct 30 04:25:52.385824 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.385733    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-maas-api-tls\") pod \"730f9801-4975-413e-98c9-e4c5b83a6c74\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") "
Oct 30 04:25:52.385908 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.385882    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-metrics-certs\") pod \"730f9801-4975-413e-98c9-e4c5b83a6c74\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") "
Oct 30 04:25:52.385968 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.385911    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730f9801-4975-413e-98c9-e4c5b83a6c74-kube-api-access-h947t\" (UniqueName: \"kubernetes.io/projected/730f9801-4975-413e-98c9-e4c5b83a6c74-kube-api-access-h947t\") pod \"730f9801-4975-413e-98c9-e4c5b83a6c74\" (UID: \"730f9801-4975-413e-98c9-e4c5b83a6c74\") "
Oct 30 04:25:52.390136 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.387730    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730f9801-4975-413e-98c9-e4c5b83a6c74-openshift-service-ca" pod "730f9801-4975-413e-98c9-e4c5b83a6c74" (UID: "730f9801-4975-413e-98c9-e4c5b83a6c74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:25:52.394474 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.394349    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-metrics-certs" pod "730f9801-4975-413e-98c9-e4c5b83a6c74" (UID: "730f9801-4975-413e-98c9-e4c5b83a6c74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:25:52.394474 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.394401    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730f9801-4975-413e-98c9-e4c5b83a6c74-kube-api-access-h947t" pod "730f9801-4975-413e-98c9-e4c5b83a6c74" (UID: "730f9801-4975-413e-98c9-e4c5b83a6c74"). InnerVolumeSpecName "kube-api-access-h947t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:52.394809 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.394791    5072 scope.go:122] "RemoveContainer" containerID="3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256"
Oct 30 04:25:52.395246 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:25:52.395163    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256\": container with ID starting with 3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256 not found: ID does not exist" containerID="3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256"
Oct 30 04:25:52.395246 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.395198    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256"} err="failed to get container status \"3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256\": rpc error: code = NotFound desc = could not find container \"3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256\": container with ID starting with 3c721798f11479e5d12a4542ca2234d0d9c140721940bf70fb35eb29a02dc256 not found: ID does not exist"
Oct 30 04:25:52.395973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.395938    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-maas-api-tls" pod "730f9801-4975-413e-98c9-e4c5b83a6c74" (UID: "730f9801-4975-413e-98c9-e4c5b83a6c74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:25:52.489205 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.489174    5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/730f9801-4975-413e-98c9-e4c5b83a6c74-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:52.489205 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.489208    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:52.489404 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.489225    5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/730f9801-4975-413e-98c9-e4c5b83a6c74-metrics-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:52.489404 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.489239    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h947t\" (UniqueName: \"kubernetes.io/projected/730f9801-4975-413e-98c9-e4c5b83a6c74-kube-api-access-h947t\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:52.721832 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.721803    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"]
Oct 30 04:25:52.734870 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:52.734832    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e3c43b-6db85cdc9c-9vwwm"]
Oct 30 04:25:54.407826 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:54.407795    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730f9801-4975-413e-98c9-e4c5b83a6c74" path="/var/lib/kubelet/pods/730f9801-4975-413e-98c9-e4c5b83a6c74/volumes"
Oct 30 04:25:56.901373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:56.901313    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:25:57.050514 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.050487    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-kube-api-access-zkxpp\" (UniqueName: \"kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-kube-api-access-zkxpp\") pod \"1e5e829f-493e-46f1-84de-c77c944d1108\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") "
Oct 30 04:25:57.050673 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.050581    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-data\") pod \"1e5e829f-493e-46f1-84de-c77c944d1108\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") "
Oct 30 04:25:57.050673 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.050608    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5e829f-493e-46f1-84de-c77c944d1108-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5e829f-493e-46f1-84de-c77c944d1108-istiod-ca-cert\") pod \"1e5e829f-493e-46f1-84de-c77c944d1108\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") "
Oct 30 04:25:57.050752 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.050690    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-socket\") pod \"1e5e829f-493e-46f1-84de-c77c944d1108\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") "
Oct 30 04:25:57.050752 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.050735    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-envoy\") pod \"1e5e829f-493e-46f1-84de-c77c944d1108\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") "
Oct 30 04:25:57.050829 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.050780    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-certs\") pod \"1e5e829f-493e-46f1-84de-c77c944d1108\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") "
Oct 30 04:25:57.050829 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.050821    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e5e829f-493e-46f1-84de-c77c944d1108-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5e829f-493e-46f1-84de-c77c944d1108-istio-podinfo\") pod \"1e5e829f-493e-46f1-84de-c77c944d1108\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") "
Oct 30 04:25:57.050953 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.050891    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-data" pod "1e5e829f-493e-46f1-84de-c77c944d1108" (UID: "1e5e829f-493e-46f1-84de-c77c944d1108"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:57.051017 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.050902    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-credential-socket\") pod \"1e5e829f-493e-46f1-84de-c77c944d1108\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") "
Oct 30 04:25:57.051017 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.050978    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-socket" pod "1e5e829f-493e-46f1-84de-c77c944d1108" (UID: "1e5e829f-493e-46f1-84de-c77c944d1108"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:57.051170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.051013    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-istio-token\" (UniqueName: \"kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-istio-token\") pod \"1e5e829f-493e-46f1-84de-c77c944d1108\" (UID: \"1e5e829f-493e-46f1-84de-c77c944d1108\") "
Oct 30 04:25:57.051170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.051138    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-credential-socket" pod "1e5e829f-493e-46f1-84de-c77c944d1108" (UID: "1e5e829f-493e-46f1-84de-c77c944d1108"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:57.051261 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.051152    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-certs" pod "1e5e829f-493e-46f1-84de-c77c944d1108" (UID: "1e5e829f-493e-46f1-84de-c77c944d1108"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:57.051375 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.051359    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5e829f-493e-46f1-84de-c77c944d1108-istiod-ca-cert" pod "1e5e829f-493e-46f1-84de-c77c944d1108" (UID: "1e5e829f-493e-46f1-84de-c77c944d1108"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:25:57.051822 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.051793    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:57.051950 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.051825    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:57.051950 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.051842    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:57.051950 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.051855    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:57.051950 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.051870    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5e829f-493e-46f1-84de-c77c944d1108-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:57.053026 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.052999    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-kube-api-access-zkxpp" pod "1e5e829f-493e-46f1-84de-c77c944d1108" (UID: "1e5e829f-493e-46f1-84de-c77c944d1108"). InnerVolumeSpecName "kube-api-access-zkxpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:57.053145 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.053118    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-envoy" pod "1e5e829f-493e-46f1-84de-c77c944d1108" (UID: "1e5e829f-493e-46f1-84de-c77c944d1108"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:57.053436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.053410    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e5e829f-493e-46f1-84de-c77c944d1108-istio-podinfo" pod "1e5e829f-493e-46f1-84de-c77c944d1108" (UID: "1e5e829f-493e-46f1-84de-c77c944d1108"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:25:57.053528 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.053503    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-istio-token" pod "1e5e829f-493e-46f1-84de-c77c944d1108" (UID: "1e5e829f-493e-46f1-84de-c77c944d1108"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:57.153600 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.153523    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5e829f-493e-46f1-84de-c77c944d1108-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:57.153600 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.153552    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5e829f-493e-46f1-84de-c77c944d1108-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:57.153600 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.153560    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:57.153600 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.153568    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkxpp\" (UniqueName: \"kubernetes.io/projected/1e5e829f-493e-46f1-84de-c77c944d1108-kube-api-access-zkxpp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:25:57.416780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.416696    5072 generic.go:356] "Generic (PLEG): container finished" podID="1e5e829f-493e-46f1-84de-c77c944d1108" containerID="7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614" exitCode=0
Oct 30 04:25:57.416780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.416763    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"
Oct 30 04:25:57.416780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.416767    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7" event={"ID":"1e5e829f-493e-46f1-84de-c77c944d1108","Type":"ContainerDied","Data":"7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614"}
Oct 30 04:25:57.417048 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.416819    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7" event={"ID":"1e5e829f-493e-46f1-84de-c77c944d1108","Type":"ContainerDied","Data":"8db1a541434e4fcf14d5566e93c0c1992315d7cc3f74026ea952b0bd68ab109a"}
Oct 30 04:25:57.417048 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.416843    5072 scope.go:122] "RemoveContainer" containerID="7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614"
Oct 30 04:25:57.428407 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.428391    5072 scope.go:122] "RemoveContainer" containerID="7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614"
Oct 30 04:25:57.428708 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:25:57.428690    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614\": container with ID starting with 7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614 not found: ID does not exist" containerID="7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614"
Oct 30 04:25:57.428754 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.428718    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614"} err="failed to get container status \"7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614\": rpc error: code = NotFound desc = could not find container \"7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614\": container with ID starting with 7104e8f4de144738e4494212c8d932ef3321578b4aaba156660cfb41045c0614 not found: ID does not exist"
Oct 30 04:25:57.483612 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.483583    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"]
Oct 30 04:25:57.496421 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:57.496397    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-qlkn7"]
Oct 30 04:25:58.416793 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:25:58.416752    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5e829f-493e-46f1-84de-c77c944d1108" path="/var/lib/kubelet/pods/1e5e829f-493e-46f1-84de-c77c944d1108/volumes"
Oct 30 04:26:00.702114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:00.702064    5072 scope.go:122] "RemoveContainer" containerID="307904a5a078e22664ab5820298f02346ebb1d14baa7c62ddcc9109869cff462"
Oct 30 04:26:10.386360 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.386273    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"]
Oct 30 04:26:10.386896 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.386534    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9" podUID="4dd61d11-85e2-499d-af65-dfd7fdedcf53" containerName="maas-api" containerID="cri-o://8c333b35bf7d49a10dcebf7ac5406fcd53d02ed6828511424ebb6c23ee3995d5" gracePeriod=30
Oct 30 04:26:10.518273 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.518239    5072 generic.go:356] "Generic (PLEG): container finished" podID="4dd61d11-85e2-499d-af65-dfd7fdedcf53" containerID="8c333b35bf7d49a10dcebf7ac5406fcd53d02ed6828511424ebb6c23ee3995d5" exitCode=0
Oct 30 04:26:10.518413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.518279    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9" event={"ID":"4dd61d11-85e2-499d-af65-dfd7fdedcf53","Type":"ContainerDied","Data":"8c333b35bf7d49a10dcebf7ac5406fcd53d02ed6828511424ebb6c23ee3995d5"}
Oct 30 04:26:10.718118 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.715292    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:26:10.821502 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.821472    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd61d11-85e2-499d-af65-dfd7fdedcf53-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd61d11-85e2-499d-af65-dfd7fdedcf53-openshift-service-ca\") pod \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") "
Oct 30 04:26:10.821766 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.821524    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-maas-api-tls\") pod \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") "
Oct 30 04:26:10.821766 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.821655    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-metrics-certs\") pod \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") "
Oct 30 04:26:10.821766 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.821730    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd61d11-85e2-499d-af65-dfd7fdedcf53-kube-api-access-cw768\" (UniqueName: \"kubernetes.io/projected/4dd61d11-85e2-499d-af65-dfd7fdedcf53-kube-api-access-cw768\") pod \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\" (UID: \"4dd61d11-85e2-499d-af65-dfd7fdedcf53\") "
Oct 30 04:26:10.821926 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.821899    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd61d11-85e2-499d-af65-dfd7fdedcf53-openshift-service-ca" pod "4dd61d11-85e2-499d-af65-dfd7fdedcf53" (UID: "4dd61d11-85e2-499d-af65-dfd7fdedcf53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:26:10.822276 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.822256    5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd61d11-85e2-499d-af65-dfd7fdedcf53-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:10.823956 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.823931    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-metrics-certs" pod "4dd61d11-85e2-499d-af65-dfd7fdedcf53" (UID: "4dd61d11-85e2-499d-af65-dfd7fdedcf53"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:26:10.824062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.823975    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-maas-api-tls" pod "4dd61d11-85e2-499d-af65-dfd7fdedcf53" (UID: "4dd61d11-85e2-499d-af65-dfd7fdedcf53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:26:10.824062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.823986    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd61d11-85e2-499d-af65-dfd7fdedcf53-kube-api-access-cw768" pod "4dd61d11-85e2-499d-af65-dfd7fdedcf53" (UID: "4dd61d11-85e2-499d-af65-dfd7fdedcf53"). InnerVolumeSpecName "kube-api-access-cw768". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:10.923928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.923820    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:10.923928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.923860    5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd61d11-85e2-499d-af65-dfd7fdedcf53-metrics-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:10.923928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:10.923875    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw768\" (UniqueName: \"kubernetes.io/projected/4dd61d11-85e2-499d-af65-dfd7fdedcf53-kube-api-access-cw768\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:11.526023 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:11.525985    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9" event={"ID":"4dd61d11-85e2-499d-af65-dfd7fdedcf53","Type":"ContainerDied","Data":"8545ea98836454fa3650da05c99da324ac99a92a39f5f6d3dc0a64fe8b0ffe05"}
Oct 30 04:26:11.526530 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:11.526060    5072 scope.go:122] "RemoveContainer" containerID="8c333b35bf7d49a10dcebf7ac5406fcd53d02ed6828511424ebb6c23ee3995d5"
Oct 30 04:26:11.526530 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:11.525997    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"
Oct 30 04:26:11.582577 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:11.582548    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"]
Oct 30 04:26:11.595669 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:11.595636    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-397bda-5677f66fd6-kkdg9"]
Oct 30 04:26:11.751099 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:26:11.751050    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:26:12.408348 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:12.408317    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd61d11-85e2-499d-af65-dfd7fdedcf53" path="/var/lib/kubelet/pods/4dd61d11-85e2-499d-af65-dfd7fdedcf53/volumes"
Oct 30 04:26:22.182165 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:22.182112    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"]
Oct 30 04:26:22.182668 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:22.182323    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd" podUID="535f2452-a76b-4861-87b1-0a278f03f35a" containerName="istio-proxy" containerID="cri-o://3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742" gracePeriod=30
Oct 30 04:26:24.247795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:24.247723    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"]
Oct 30 04:26:24.248323 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:24.248108    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs" podUID="11a83356-8ef8-4cc7-a0e9-47015986597f" containerName="istio-proxy" containerID="cri-o://04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4" gracePeriod=30
Oct 30 04:26:24.261004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:24.260969    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"]
Oct 30 04:26:24.261328 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:24.261274    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22" podUID="de013201-a8b0-4d48-b0f3-d17fc1e9d952" containerName="istio-proxy" containerID="cri-o://30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a" gracePeriod=30
Oct 30 04:26:27.472394 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.472369    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:26:27.622213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622111    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-socket\") pod \"535f2452-a76b-4861-87b1-0a278f03f35a\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") "
Oct 30 04:26:27.622213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622208    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/535f2452-a76b-4861-87b1-0a278f03f35a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535f2452-a76b-4861-87b1-0a278f03f35a-istio-podinfo\") pod \"535f2452-a76b-4861-87b1-0a278f03f35a\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") "
Oct 30 04:26:27.622454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622235    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-istio-token\" (UniqueName: \"kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-istio-token\") pod \"535f2452-a76b-4861-87b1-0a278f03f35a\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") "
Oct 30 04:26:27.622454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622289    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-data\") pod \"535f2452-a76b-4861-87b1-0a278f03f35a\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") "
Oct 30 04:26:27.622454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622311    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-certs\") pod \"535f2452-a76b-4861-87b1-0a278f03f35a\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") "
Oct 30 04:26:27.622454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622363    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-envoy\") pod \"535f2452-a76b-4861-87b1-0a278f03f35a\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") "
Oct 30 04:26:27.622454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622401    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-socket" pod "535f2452-a76b-4861-87b1-0a278f03f35a" (UID: "535f2452-a76b-4861-87b1-0a278f03f35a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:27.622454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622419    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-kube-api-access-fkhjj\" (UniqueName: \"kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-kube-api-access-fkhjj\") pod \"535f2452-a76b-4861-87b1-0a278f03f35a\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") "
Oct 30 04:26:27.622454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622442    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/535f2452-a76b-4861-87b1-0a278f03f35a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535f2452-a76b-4861-87b1-0a278f03f35a-istiod-ca-cert\") pod \"535f2452-a76b-4861-87b1-0a278f03f35a\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") "
Oct 30 04:26:27.622845 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622553    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-credential-socket\") pod \"535f2452-a76b-4861-87b1-0a278f03f35a\" (UID: \"535f2452-a76b-4861-87b1-0a278f03f35a\") "
Oct 30 04:26:27.622845 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622578    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-data" pod "535f2452-a76b-4861-87b1-0a278f03f35a" (UID: "535f2452-a76b-4861-87b1-0a278f03f35a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:27.622953 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622838    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535f2452-a76b-4861-87b1-0a278f03f35a-istiod-ca-cert" pod "535f2452-a76b-4861-87b1-0a278f03f35a" (UID: "535f2452-a76b-4861-87b1-0a278f03f35a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:26:27.623107 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.622949    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-certs" pod "535f2452-a76b-4861-87b1-0a278f03f35a" (UID: "535f2452-a76b-4861-87b1-0a278f03f35a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:27.623107 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.623051    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-credential-socket" pod "535f2452-a76b-4861-87b1-0a278f03f35a" (UID: "535f2452-a76b-4861-87b1-0a278f03f35a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:27.623382 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.623363    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:27.623445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.623382    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:27.623445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.623392    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535f2452-a76b-4861-87b1-0a278f03f35a-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:27.623445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.623400    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:27.623445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.623409    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:27.624751 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.624721    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/535f2452-a76b-4861-87b1-0a278f03f35a-istio-podinfo" pod "535f2452-a76b-4861-87b1-0a278f03f35a" (UID: "535f2452-a76b-4861-87b1-0a278f03f35a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:26:27.624968 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.624945    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-kube-api-access-fkhjj" pod "535f2452-a76b-4861-87b1-0a278f03f35a" (UID: "535f2452-a76b-4861-87b1-0a278f03f35a"). InnerVolumeSpecName "kube-api-access-fkhjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:27.625186 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.625168    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-envoy" pod "535f2452-a76b-4861-87b1-0a278f03f35a" (UID: "535f2452-a76b-4861-87b1-0a278f03f35a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:27.625186 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.625179    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-istio-token" pod "535f2452-a76b-4861-87b1-0a278f03f35a" (UID: "535f2452-a76b-4861-87b1-0a278f03f35a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:27.656990 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.656958    5072 generic.go:356] "Generic (PLEG): container finished" podID="535f2452-a76b-4861-87b1-0a278f03f35a" containerID="3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742" exitCode=0
Oct 30 04:26:27.657216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.657010    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"
Oct 30 04:26:27.657216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.657019    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd" event={"ID":"535f2452-a76b-4861-87b1-0a278f03f35a","Type":"ContainerDied","Data":"3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742"}
Oct 30 04:26:27.657216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.657113    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd" event={"ID":"535f2452-a76b-4861-87b1-0a278f03f35a","Type":"ContainerDied","Data":"fac7956f1d881a43504127692bd5acf1e2ebfd37f0fd336b008a803be527df4c"}
Oct 30 04:26:27.657216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.657126    5072 scope.go:122] "RemoveContainer" containerID="3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742"
Oct 30 04:26:27.668122 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.668095    5072 scope.go:122] "RemoveContainer" containerID="3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742"
Oct 30 04:26:27.668422 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:26:27.668402    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742\": container with ID starting with 3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742 not found: ID does not exist" containerID="3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742"
Oct 30 04:26:27.668482 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.668431    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742"} err="failed to get container status \"3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742\": rpc error: code = NotFound desc = could not find container \"3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742\": container with ID starting with 3f38dae8061cdb5be7ad734f0a84731d7b5eca6f6f2b0d188f7f9b26cf687742 not found: ID does not exist"
Oct 30 04:26:27.711871 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.711835    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"]
Oct 30 04:26:27.715555 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.715526    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-tndqd"]
Oct 30 04:26:27.724282 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.724262    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535f2452-a76b-4861-87b1-0a278f03f35a-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:27.724282 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.724282    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:27.724425 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.724290    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535f2452-a76b-4861-87b1-0a278f03f35a-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:27.724425 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:27.724298    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkhjj\" (UniqueName: \"kubernetes.io/projected/535f2452-a76b-4861-87b1-0a278f03f35a-kube-api-access-fkhjj\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:28.411563 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:28.411530    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="535f2452-a76b-4861-87b1-0a278f03f35a" path="/var/lib/kubelet/pods/535f2452-a76b-4861-87b1-0a278f03f35a/volumes"
Oct 30 04:26:29.187795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.187763    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"]
Oct 30 04:26:29.548911 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.548879    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:26:29.578014 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.577988    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:26:29.647972 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.647937    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-kube-api-access-tqlmr\" (UniqueName: \"kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-kube-api-access-tqlmr\") pod \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") "
Oct 30 04:26:29.648213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.647993    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-envoy\") pod \"11a83356-8ef8-4cc7-a0e9-47015986597f\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") "
Oct 30 04:26:29.648213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648030    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-token\" (UniqueName: \"kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-token\") pod \"11a83356-8ef8-4cc7-a0e9-47015986597f\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") "
Oct 30 04:26:29.648213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648108    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-credential-socket\") pod \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") "
Oct 30 04:26:29.648213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648156    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-data\") pod \"11a83356-8ef8-4cc7-a0e9-47015986597f\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") "
Oct 30 04:26:29.648213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648186    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a83356-8ef8-4cc7-a0e9-47015986597f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a83356-8ef8-4cc7-a0e9-47015986597f-istiod-ca-cert\") pod \"11a83356-8ef8-4cc7-a0e9-47015986597f\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") "
Oct 30 04:26:29.648449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648226    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-socket\") pod \"11a83356-8ef8-4cc7-a0e9-47015986597f\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") "
Oct 30 04:26:29.648449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648291    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-podinfo\") pod \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") "
Oct 30 04:26:29.648449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648351    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-kube-api-access-4f6cs\" (UniqueName: \"kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-kube-api-access-4f6cs\") pod \"11a83356-8ef8-4cc7-a0e9-47015986597f\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") "
Oct 30 04:26:29.648449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648395    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-certs\") pod \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") "
Oct 30 04:26:29.648449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648429    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-envoy\") pod \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") "
Oct 30 04:26:29.648610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648481    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-certs\") pod \"11a83356-8ef8-4cc7-a0e9-47015986597f\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") "
Oct 30 04:26:29.648610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648519    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-data\") pod \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") "
Oct 30 04:26:29.648610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648553    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-podinfo\") pod \"11a83356-8ef8-4cc7-a0e9-47015986597f\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") "
Oct 30 04:26:29.648610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648604    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-socket\") pod \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") "
Oct 30 04:26:29.648729 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648661    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-credential-socket\") pod \"11a83356-8ef8-4cc7-a0e9-47015986597f\" (UID: \"11a83356-8ef8-4cc7-a0e9-47015986597f\") "
Oct 30 04:26:29.648729 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648684    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-token\" (UniqueName: \"kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-token\") pod \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") "
Oct 30 04:26:29.648789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648754    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istiod-ca-cert\") pod \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\" (UID: \"de013201-a8b0-4d48-b0f3-d17fc1e9d952\") "
Oct 30 04:26:29.649038 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.648861    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-socket" pod "11a83356-8ef8-4cc7-a0e9-47015986597f" (UID: "11a83356-8ef8-4cc7-a0e9-47015986597f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.649186 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.649093    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-credential-socket" pod "de013201-a8b0-4d48-b0f3-d17fc1e9d952" (UID: "de013201-a8b0-4d48-b0f3-d17fc1e9d952"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.650204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.649256    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-socket" pod "de013201-a8b0-4d48-b0f3-d17fc1e9d952" (UID: "de013201-a8b0-4d48-b0f3-d17fc1e9d952"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.650204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.649453    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-data" pod "de013201-a8b0-4d48-b0f3-d17fc1e9d952" (UID: "de013201-a8b0-4d48-b0f3-d17fc1e9d952"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.650204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.649388    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istiod-ca-cert" pod "de013201-a8b0-4d48-b0f3-d17fc1e9d952" (UID: "de013201-a8b0-4d48-b0f3-d17fc1e9d952"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:26:29.650204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.649473    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.650204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.649491    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.650204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.649505    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.650204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.649577    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-data" pod "11a83356-8ef8-4cc7-a0e9-47015986597f" (UID: "11a83356-8ef8-4cc7-a0e9-47015986597f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.650204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.649686    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-credential-socket" pod "11a83356-8ef8-4cc7-a0e9-47015986597f" (UID: "11a83356-8ef8-4cc7-a0e9-47015986597f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.650204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.649803    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a83356-8ef8-4cc7-a0e9-47015986597f-istiod-ca-cert" pod "11a83356-8ef8-4cc7-a0e9-47015986597f" (UID: "11a83356-8ef8-4cc7-a0e9-47015986597f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:26:29.650204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.650137    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-certs" pod "de013201-a8b0-4d48-b0f3-d17fc1e9d952" (UID: "de013201-a8b0-4d48-b0f3-d17fc1e9d952"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.650906 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.650872    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-certs" pod "11a83356-8ef8-4cc7-a0e9-47015986597f" (UID: "11a83356-8ef8-4cc7-a0e9-47015986597f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.652293 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.652257    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-podinfo" pod "de013201-a8b0-4d48-b0f3-d17fc1e9d952" (UID: "de013201-a8b0-4d48-b0f3-d17fc1e9d952"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:26:29.652409 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.652384    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-kube-api-access-4f6cs" pod "11a83356-8ef8-4cc7-a0e9-47015986597f" (UID: "11a83356-8ef8-4cc7-a0e9-47015986597f"). InnerVolumeSpecName "kube-api-access-4f6cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:29.652836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.652810    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-token" pod "de013201-a8b0-4d48-b0f3-d17fc1e9d952" (UID: "de013201-a8b0-4d48-b0f3-d17fc1e9d952"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:29.653508 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.653463    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-kube-api-access-tqlmr" pod "de013201-a8b0-4d48-b0f3-d17fc1e9d952" (UID: "de013201-a8b0-4d48-b0f3-d17fc1e9d952"). InnerVolumeSpecName "kube-api-access-tqlmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:29.653508 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.653465    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-podinfo" pod "11a83356-8ef8-4cc7-a0e9-47015986597f" (UID: "11a83356-8ef8-4cc7-a0e9-47015986597f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:26:29.654126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.654103    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-token" pod "11a83356-8ef8-4cc7-a0e9-47015986597f" (UID: "11a83356-8ef8-4cc7-a0e9-47015986597f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:29.654694 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.654672    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-envoy" pod "de013201-a8b0-4d48-b0f3-d17fc1e9d952" (UID: "de013201-a8b0-4d48-b0f3-d17fc1e9d952"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.655004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.654986    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-envoy" pod "11a83356-8ef8-4cc7-a0e9-47015986597f" (UID: "11a83356-8ef8-4cc7-a0e9-47015986597f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.686812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.686759    5072 generic.go:356] "Generic (PLEG): container finished" podID="11a83356-8ef8-4cc7-a0e9-47015986597f" containerID="04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4" exitCode=0
Oct 30 04:26:29.686995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.686839    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"
Oct 30 04:26:29.686995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.686859    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs" event={"ID":"11a83356-8ef8-4cc7-a0e9-47015986597f","Type":"ContainerDied","Data":"04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4"}
Oct 30 04:26:29.686995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.686956    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs" event={"ID":"11a83356-8ef8-4cc7-a0e9-47015986597f","Type":"ContainerDied","Data":"27d0c74e0f30250e8c8206dac393f15727dd470c51cf075a0c06ca939c3d0d1a"}
Oct 30 04:26:29.686995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.686988    5072 scope.go:122] "RemoveContainer" containerID="04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4"
Oct 30 04:26:29.689619 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.689580    5072 generic.go:356] "Generic (PLEG): container finished" podID="de013201-a8b0-4d48-b0f3-d17fc1e9d952" containerID="30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a" exitCode=0
Oct 30 04:26:29.689753 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.689652    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22" event={"ID":"de013201-a8b0-4d48-b0f3-d17fc1e9d952","Type":"ContainerDied","Data":"30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a"}
Oct 30 04:26:29.689753 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.689697    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22" event={"ID":"de013201-a8b0-4d48-b0f3-d17fc1e9d952","Type":"ContainerDied","Data":"bcb52d21fef1f3c33a9cc6a82ba96360c6c70e847a2b91fd529b7b77614b2fb6"}
Oct 30 04:26:29.689877 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.689786    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"
Oct 30 04:26:29.702278 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.702253    5072 scope.go:122] "RemoveContainer" containerID="04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4"
Oct 30 04:26:29.703584 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:26:29.703535    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4\": container with ID starting with 04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4 not found: ID does not exist" containerID="04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4"
Oct 30 04:26:29.703712 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.703575    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4"} err="failed to get container status \"04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4\": rpc error: code = NotFound desc = could not find container \"04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4\": container with ID starting with 04509d252056cd0fe21323383f21a758e32c24bb316b86d9a0d2798340255cb4 not found: ID does not exist"
Oct 30 04:26:29.703712 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.703601    5072 scope.go:122] "RemoveContainer" containerID="30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a"
Oct 30 04:26:29.713569 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.713551    5072 scope.go:122] "RemoveContainer" containerID="30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a"
Oct 30 04:26:29.714008 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:26:29.713977    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a\": container with ID starting with 30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a not found: ID does not exist" containerID="30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a"
Oct 30 04:26:29.714238 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.714017    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a"} err="failed to get container status \"30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a\": rpc error: code = NotFound desc = could not find container \"30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a\": container with ID starting with 30fb01fff30b032644ac5dc9c53be443907703fc12e5c0bf68361e983f2faf7a not found: ID does not exist"
Oct 30 04:26:29.750541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750455    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750484    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqlmr\" (UniqueName: \"kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-kube-api-access-tqlmr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750495    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750504    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750513    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750521    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a83356-8ef8-4cc7-a0e9-47015986597f-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750529    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750537    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f6cs\" (UniqueName: \"kubernetes.io/projected/11a83356-8ef8-4cc7-a0e9-47015986597f-kube-api-access-4f6cs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750545    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750553    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750562    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750569    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750576    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a83356-8ef8-4cc7-a0e9-47015986597f-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750584    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a83356-8ef8-4cc7-a0e9-47015986597f-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.750931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.750591    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de013201-a8b0-4d48-b0f3-d17fc1e9d952-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.762976 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.762945    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"]
Oct 30 04:26:29.766662 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.766639    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-8rmrs"]
Oct 30 04:26:29.799893 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.799741    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"]
Oct 30 04:26:29.802812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:29.802787    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-4ht22"]
Oct 30 04:26:30.407533 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:30.407504    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a83356-8ef8-4cc7-a0e9-47015986597f" path="/var/lib/kubelet/pods/11a83356-8ef8-4cc7-a0e9-47015986597f/volumes"
Oct 30 04:26:30.407979 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:30.407965    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de013201-a8b0-4d48-b0f3-d17fc1e9d952" path="/var/lib/kubelet/pods/de013201-a8b0-4d48-b0f3-d17fc1e9d952/volumes"
Oct 30 04:26:41.752670 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:26:41.752634    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:26:44.211805 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.211764    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt" podUID="ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1" containerName="main" containerID="cri-o://a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb" gracePeriod=45
Oct 30 04:26:44.498230 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.498202    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:26:44.636161 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.636123    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-model-cache\") pod \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") "
Oct 30 04:26:44.636339 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.636225    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-dshm\") pod \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") "
Oct 30 04:26:44.636339 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.636252    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-kube-api-access-hdhgm\" (UniqueName: \"kubernetes.io/projected/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-kube-api-access-hdhgm\") pod \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") "
Oct 30 04:26:44.636339 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.636301    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tmp-dir\") pod \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") "
Oct 30 04:26:44.636532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.636350    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tls-certs\") pod \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") "
Oct 30 04:26:44.636532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.636379    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-home\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-home\") pod \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\" (UID: \"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1\") "
Oct 30 04:26:44.636532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.636469    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-model-cache" pod "ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1" (UID: "ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:44.636727 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.636701    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-home" pod "ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1" (UID: "ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:44.636931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.636912    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:44.637096 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.636936    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:44.637096 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.636934    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tmp-dir" pod "ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1" (UID: "ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:44.638843 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.638815    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tls-certs" pod "ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1" (UID: "ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:26:44.639014 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.638995    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-dshm" pod "ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1" (UID: "ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:44.639014 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.639003    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-kube-api-access-hdhgm" pod "ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1" (UID: "ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1"). InnerVolumeSpecName "kube-api-access-hdhgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:44.738722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.738639    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:44.738722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.738669    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdhgm\" (UniqueName: \"kubernetes.io/projected/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-kube-api-access-hdhgm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:44.738722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.738680    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:44.738722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.738689    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:26:44.787022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.786989    5072 generic.go:356] "Generic (PLEG): container finished" podID="ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1" containerID="a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb" exitCode=0
Oct 30 04:26:44.787209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.787032    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"
Oct 30 04:26:44.787209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.787054    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt" event={"ID":"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1","Type":"ContainerDied","Data":"a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb"}
Oct 30 04:26:44.787209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.787107    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt" event={"ID":"ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1","Type":"ContainerDied","Data":"d4cd7238ed7c8e728eb0eb1d2f588afa9613d7674dd0ceea7309832ffeae4974"}
Oct 30 04:26:44.787209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.787131    5072 scope.go:122] "RemoveContainer" containerID="a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb"
Oct 30 04:26:44.800979 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.800951    5072 scope.go:122] "RemoveContainer" containerID="a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb"
Oct 30 04:26:44.801577 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:26:44.801554    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb\": container with ID starting with a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb not found: ID does not exist" containerID="a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb"
Oct 30 04:26:44.801668 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.801588    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb"} err="failed to get container status \"a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb\": rpc error: code = NotFound desc = could not find container \"a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb\": container with ID starting with a8e130579237e00f8835285fdd19f4f483e7662e5a4a97c36557467bcaa6fafb not found: ID does not exist"
Oct 30 04:26:44.846738 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.846706    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"]
Oct 30 04:26:44.859197 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:44.859161    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1a0d3427/e2e-lifecycle-model-1a0d3427-kserve-765d9699b5-jngzt"]
Oct 30 04:26:46.407740 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:26:46.407707    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1" path="/var/lib/kubelet/pods/ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1/volumes"
Oct 30 04:27:00.628905 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:00.628872    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:27:00.635761 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:00.635740    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:27:01.527413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.527384    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"]
Oct 30 04:27:01.528009 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.527993    5072 cpu_manager.go:414] "containerMap: removing container" podUID="94072976-07f9-412d-bf5c-674d66ad42c8" containerName="maas-api"
Oct 30 04:27:01.528114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528011    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94072976-07f9-412d-bf5c-674d66ad42c8" containerName="maas-api"
Oct 30 04:27:01.528114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528023    5072 cpu_manager.go:414] "containerMap: removing container" podUID="11a83356-8ef8-4cc7-a0e9-47015986597f" containerName="istio-proxy"
Oct 30 04:27:01.528114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528028    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a83356-8ef8-4cc7-a0e9-47015986597f" containerName="istio-proxy"
Oct 30 04:27:01.528114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528041    5072 cpu_manager.go:414] "containerMap: removing container" podUID="730f9801-4975-413e-98c9-e4c5b83a6c74" containerName="maas-api"
Oct 30 04:27:01.528114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528046    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730f9801-4975-413e-98c9-e4c5b83a6c74" containerName="maas-api"
Oct 30 04:27:01.528114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528055    5072 cpu_manager.go:414] "containerMap: removing container" podUID="4dd61d11-85e2-499d-af65-dfd7fdedcf53" containerName="maas-api"
Oct 30 04:27:01.528114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528060    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd61d11-85e2-499d-af65-dfd7fdedcf53" containerName="maas-api"
Oct 30 04:27:01.528114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528071    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1" containerName="main"
Oct 30 04:27:01.528114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528093    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1" containerName="main"
Oct 30 04:27:01.528114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528106    5072 cpu_manager.go:414] "containerMap: removing container" podUID="de013201-a8b0-4d48-b0f3-d17fc1e9d952" containerName="istio-proxy"
Oct 30 04:27:01.528114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528111    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de013201-a8b0-4d48-b0f3-d17fc1e9d952" containerName="istio-proxy"
Oct 30 04:27:01.528114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528120    5072 cpu_manager.go:414] "containerMap: removing container" podUID="535f2452-a76b-4861-87b1-0a278f03f35a" containerName="istio-proxy"
Oct 30 04:27:01.528544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528125    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="535f2452-a76b-4861-87b1-0a278f03f35a" containerName="istio-proxy"
Oct 30 04:27:01.528544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528132    5072 cpu_manager.go:414] "containerMap: removing container" podUID="1e5e829f-493e-46f1-84de-c77c944d1108" containerName="istio-proxy"
Oct 30 04:27:01.528544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528138    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5e829f-493e-46f1-84de-c77c944d1108" containerName="istio-proxy"
Oct 30 04:27:01.528544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528226    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="730f9801-4975-413e-98c9-e4c5b83a6c74" containerName="maas-api"
Oct 30 04:27:01.528544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528236    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd61d11-85e2-499d-af65-dfd7fdedcf53" containerName="maas-api"
Oct 30 04:27:01.528544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528243    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef61e149-c7e9-4ea8-ab94-82cd19fdd4b1" containerName="main"
Oct 30 04:27:01.528544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528249    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="de013201-a8b0-4d48-b0f3-d17fc1e9d952" containerName="istio-proxy"
Oct 30 04:27:01.528544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528256    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="535f2452-a76b-4861-87b1-0a278f03f35a" containerName="istio-proxy"
Oct 30 04:27:01.528544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528264    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5e829f-493e-46f1-84de-c77c944d1108" containerName="istio-proxy"
Oct 30 04:27:01.528544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528274    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="94072976-07f9-412d-bf5c-674d66ad42c8" containerName="maas-api"
Oct 30 04:27:01.528544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.528279    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a83356-8ef8-4cc7-a0e9-47015986597f" containerName="istio-proxy"
Oct 30 04:27:01.533140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.533123    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"
Oct 30 04:27:01.561155 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.561129    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf4238\""
Oct 30 04:27:01.561291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.561241    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf4238-dockercfg-hfmp6\""
Oct 30 04:27:01.565220 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.565194    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"]
Oct 30 04:27:01.639592 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.639563    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd1e079-8a0d-4aac-98a1-33347d2d515a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4\" (UID: \"5fd1e079-8a0d-4aac-98a1-33347d2d515a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"
Oct 30 04:27:01.639929 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.639612    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhz2b\" (UniqueName: \"kubernetes.io/projected/5fd1e079-8a0d-4aac-98a1-33347d2d515a-kube-api-access-dhz2b\") pod \"payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4\" (UID: \"5fd1e079-8a0d-4aac-98a1-33347d2d515a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"
Oct 30 04:27:01.740671 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.740632    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhz2b\" (UniqueName: \"kubernetes.io/projected/5fd1e079-8a0d-4aac-98a1-33347d2d515a-kube-api-access-dhz2b\") pod \"payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4\" (UID: \"5fd1e079-8a0d-4aac-98a1-33347d2d515a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"
Oct 30 04:27:01.740860 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.740837    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd1e079-8a0d-4aac-98a1-33347d2d515a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4\" (UID: \"5fd1e079-8a0d-4aac-98a1-33347d2d515a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"
Oct 30 04:27:01.741527 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.741509    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd1e079-8a0d-4aac-98a1-33347d2d515a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4\" (UID: \"5fd1e079-8a0d-4aac-98a1-33347d2d515a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"
Oct 30 04:27:01.765299 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.765269    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhz2b\" (UniqueName: \"kubernetes.io/projected/5fd1e079-8a0d-4aac-98a1-33347d2d515a-kube-api-access-dhz2b\") pod \"payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4\" (UID: \"5fd1e079-8a0d-4aac-98a1-33347d2d515a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"
Oct 30 04:27:01.847050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:01.846962    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"
Oct 30 04:27:02.148282 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:02.148260    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"]
Oct 30 04:27:02.149784 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:27:02.149756    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd1e079_8a0d_4aac_98a1_33347d2d515a.slice/crio-b21a1052ac3d05fd9c758efc936a858a7a2bdeed32ba7754b6189ede5851678a WatchSource:0}: Error finding container b21a1052ac3d05fd9c758efc936a858a7a2bdeed32ba7754b6189ede5851678a: Status 404 returned error can't find the container with id b21a1052ac3d05fd9c758efc936a858a7a2bdeed32ba7754b6189ede5851678a
Oct 30 04:27:02.897971 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:02.897936    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4" event={"ID":"5fd1e079-8a0d-4aac-98a1-33347d2d515a","Type":"ContainerStarted","Data":"b21a1052ac3d05fd9c758efc936a858a7a2bdeed32ba7754b6189ede5851678a"}
Oct 30 04:27:03.906915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:03.906879    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4" event={"ID":"5fd1e079-8a0d-4aac-98a1-33347d2d515a","Type":"ContainerStarted","Data":"16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393"}
Oct 30 04:27:03.907448 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:03.906979    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"
Oct 30 04:27:03.952852 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:03.952796    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4" podStartSLOduration=1.941394484 podStartE2EDuration="2.952782516s" totalImagesPullingTime="1.011388032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:03.950486441 +0000 UTC m=+1803.691909539" watchObservedRunningTime="2026-10-30 04:27:03.952782516 +0000 UTC m=+1803.694205612"
Oct 30 04:27:11.123275 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.123242    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"]
Oct 30 04:27:11.127540 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.127523    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"
Oct 30 04:27:11.158252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.158223    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6dcea5-dockercfg-vk549\""
Oct 30 04:27:11.158252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.158218    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6dcea5\""
Oct 30 04:27:11.161072 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.161044    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"]
Oct 30 04:27:11.253151 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.253115    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b1201f-aea5-4193-84fd-a9c8a1b2779d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm\" (UID: \"64b1201f-aea5-4193-84fd-a9c8a1b2779d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"
Oct 30 04:27:11.253331 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.253235    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxt2\" (UniqueName: \"kubernetes.io/projected/64b1201f-aea5-4193-84fd-a9c8a1b2779d-kube-api-access-rfxt2\") pod \"payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm\" (UID: \"64b1201f-aea5-4193-84fd-a9c8a1b2779d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"
Oct 30 04:27:11.354604 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.354566    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxt2\" (UniqueName: \"kubernetes.io/projected/64b1201f-aea5-4193-84fd-a9c8a1b2779d-kube-api-access-rfxt2\") pod \"payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm\" (UID: \"64b1201f-aea5-4193-84fd-a9c8a1b2779d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"
Oct 30 04:27:11.354772 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.354701    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b1201f-aea5-4193-84fd-a9c8a1b2779d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm\" (UID: \"64b1201f-aea5-4193-84fd-a9c8a1b2779d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"
Oct 30 04:27:11.355410 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.355390    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b1201f-aea5-4193-84fd-a9c8a1b2779d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm\" (UID: \"64b1201f-aea5-4193-84fd-a9c8a1b2779d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"
Oct 30 04:27:11.379041 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.378979    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxt2\" (UniqueName: \"kubernetes.io/projected/64b1201f-aea5-4193-84fd-a9c8a1b2779d-kube-api-access-rfxt2\") pod \"payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm\" (UID: \"64b1201f-aea5-4193-84fd-a9c8a1b2779d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"
Oct 30 04:27:11.441679 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.441644    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"
Oct 30 04:27:11.752611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.752582    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"]
Oct 30 04:27:11.771036 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:27:11.771004    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:27:11.957036 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.956999    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm" event={"ID":"64b1201f-aea5-4193-84fd-a9c8a1b2779d","Type":"ContainerStarted","Data":"085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb"}
Oct 30 04:27:11.957281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.957162    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"
Oct 30 04:27:11.957281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:11.957192    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm" event={"ID":"64b1201f-aea5-4193-84fd-a9c8a1b2779d","Type":"ContainerStarted","Data":"a72cb0318d0cf8fabc98a7e1f787e54f0b5a93719872af8e5534439ab262f635"}
Oct 30 04:27:12.002452 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:12.002347    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm" podStartSLOduration=1.002332931 podStartE2EDuration="1.002332931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:12.000195536 +0000 UTC m=+1811.741618672" watchObservedRunningTime="2026-10-30 04:27:12.002332931 +0000 UTC m=+1811.743756027"
Oct 30 04:27:14.915232 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:14.915205    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"
Oct 30 04:27:17.386118 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:17.386069    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"]
Oct 30 04:27:17.390667 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:17.390644    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"
Oct 30 04:27:17.409913 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:17.409887    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"]
Oct 30 04:27:17.419808 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:17.419780    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-baa474\""
Oct 30 04:27:17.419932 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:17.419812    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-baa474-dockercfg-dgwzk\""
Oct 30 04:27:17.526610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:17.526574    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9\" (UID: \"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"
Oct 30 04:27:17.526802 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:17.526708    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6rqd\" (UniqueName: \"kubernetes.io/projected/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-kube-api-access-z6rqd\") pod \"payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9\" (UID: \"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"
Oct 30 04:27:17.628309 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:17.628271    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6rqd\" (UniqueName: \"kubernetes.io/projected/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-kube-api-access-z6rqd\") pod \"payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9\" (UID: \"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"
Oct 30 04:27:17.628474 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:17.628426    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9\" (UID: \"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"
Oct 30 04:27:17.629158 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:17.629135    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9\" (UID: \"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"
Oct 30 04:27:17.652747 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:17.652689    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6rqd\" (UniqueName: \"kubernetes.io/projected/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-kube-api-access-z6rqd\") pod \"payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9\" (UID: \"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"
Oct 30 04:27:17.706345 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:17.706315    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"
Oct 30 04:27:18.010508 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:27:18.010480    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ecbf17_3341_4e24_9aab_b88cb1bd10e9.slice/crio-ab5227831b0ee1b47bb39ae19c3631fe93feedb99eb45e0ad9cd5b87e7603093 WatchSource:0}: Error finding container ab5227831b0ee1b47bb39ae19c3631fe93feedb99eb45e0ad9cd5b87e7603093: Status 404 returned error can't find the container with id ab5227831b0ee1b47bb39ae19c3631fe93feedb99eb45e0ad9cd5b87e7603093
Oct 30 04:27:18.010645 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:18.010522    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"]
Oct 30 04:27:19.000944 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:19.000907    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9" event={"ID":"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9","Type":"ContainerStarted","Data":"9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c"}
Oct 30 04:27:19.001462 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:19.000964    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9" event={"ID":"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9","Type":"ContainerStarted","Data":"ab5227831b0ee1b47bb39ae19c3631fe93feedb99eb45e0ad9cd5b87e7603093"}
Oct 30 04:27:19.001462 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:19.001028    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"
Oct 30 04:27:19.044939 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:19.044893    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9" podStartSLOduration=2.044879796 podStartE2EDuration="2.044879796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:19.042355777 +0000 UTC m=+1818.783778874" watchObservedRunningTime="2026-10-30 04:27:19.044879796 +0000 UTC m=+1818.786302893"
Oct 30 04:27:22.965987 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:22.965956    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"
Oct 30 04:27:30.009598 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:27:30.009568    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"
Oct 30 04:27:41.763637 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:27:41.763549    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:28:06.062561 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.062524    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"]
Oct 30 04:28:06.066935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.066907    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:06.086794 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.086763    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"]
Oct 30 04:28:06.095035 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.095002    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:28:06.095175 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.095046    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:28:06.177340 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.177311    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6aa08f-kzqtt\" (UID: \"8cf49544-93d5-40fa-9f12-5d56e15610d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:06.177539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.177352    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vsjw\" (UniqueName: \"kubernetes.io/projected/8cf49544-93d5-40fa-9f12-5d56e15610d9-kube-api-access-6vsjw\") pod \"maas-api-revoke-sub-9d6aa08f-kzqtt\" (UID: \"8cf49544-93d5-40fa-9f12-5d56e15610d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:06.177600 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.177532    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6aa08f-kzqtt\" (UID: \"8cf49544-93d5-40fa-9f12-5d56e15610d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:06.278277 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.278244    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6aa08f-kzqtt\" (UID: \"8cf49544-93d5-40fa-9f12-5d56e15610d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:06.278446 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.278294    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vsjw\" (UniqueName: \"kubernetes.io/projected/8cf49544-93d5-40fa-9f12-5d56e15610d9-kube-api-access-6vsjw\") pod \"maas-api-revoke-sub-9d6aa08f-kzqtt\" (UID: \"8cf49544-93d5-40fa-9f12-5d56e15610d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:06.278446 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.278422    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6aa08f-kzqtt\" (UID: \"8cf49544-93d5-40fa-9f12-5d56e15610d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:06.278937 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.278909    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6aa08f-kzqtt\" (UID: \"8cf49544-93d5-40fa-9f12-5d56e15610d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:06.281060 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.281035    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6aa08f-kzqtt\" (UID: \"8cf49544-93d5-40fa-9f12-5d56e15610d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:06.305725 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.305697    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vsjw\" (UniqueName: \"kubernetes.io/projected/8cf49544-93d5-40fa-9f12-5d56e15610d9-kube-api-access-6vsjw\") pod \"maas-api-revoke-sub-9d6aa08f-kzqtt\" (UID: \"8cf49544-93d5-40fa-9f12-5d56e15610d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:06.384531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.384460    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:06.688601 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:06.688564    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"]
Oct 30 04:28:06.693434 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:28:06.692664    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf49544_93d5_40fa_9f12_5d56e15610d9.slice/crio-030dc0736f7cbe72ffe0402f61b29e1a836aece761b35ea7bf140e1e1b4259b0 WatchSource:0}: Error finding container 030dc0736f7cbe72ffe0402f61b29e1a836aece761b35ea7bf140e1e1b4259b0: Status 404 returned error can't find the container with id 030dc0736f7cbe72ffe0402f61b29e1a836aece761b35ea7bf140e1e1b4259b0
Oct 30 04:28:07.334436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:07.334348    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt" event={"ID":"8cf49544-93d5-40fa-9f12-5d56e15610d9","Type":"ContainerStarted","Data":"423874d3683691442230f845607aa9f44e9dc4dcee9a8653fac2e0dc4a6bfeb2"}
Oct 30 04:28:07.334436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:07.334418    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt" event={"ID":"8cf49544-93d5-40fa-9f12-5d56e15610d9","Type":"ContainerStarted","Data":"030dc0736f7cbe72ffe0402f61b29e1a836aece761b35ea7bf140e1e1b4259b0"}
Oct 30 04:28:07.376506 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:07.376459    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt" podStartSLOduration=1.376444842 podStartE2EDuration="1.376444842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:07.375149911 +0000 UTC m=+1867.116573005" watchObservedRunningTime="2026-10-30 04:28:07.376444842 +0000 UTC m=+1867.117867939"
Oct 30 04:28:08.341747 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:08.341712    5072 generic.go:356] "Generic (PLEG): container finished" podID="8cf49544-93d5-40fa-9f12-5d56e15610d9" containerID="423874d3683691442230f845607aa9f44e9dc4dcee9a8653fac2e0dc4a6bfeb2" exitCode=0
Oct 30 04:28:08.342215 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:08.341806    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt" event={"ID":"8cf49544-93d5-40fa-9f12-5d56e15610d9","Type":"ContainerDied","Data":"423874d3683691442230f845607aa9f44e9dc4dcee9a8653fac2e0dc4a6bfeb2"}
Oct 30 04:28:09.501883 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:09.501859    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:09.512430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:09.512406    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-xks-ca\") pod \"8cf49544-93d5-40fa-9f12-5d56e15610d9\" (UID: \"8cf49544-93d5-40fa-9f12-5d56e15610d9\") "
Oct 30 04:28:09.512585 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:09.512446    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf49544-93d5-40fa-9f12-5d56e15610d9-kube-api-access-6vsjw\" (UniqueName: \"kubernetes.io/projected/8cf49544-93d5-40fa-9f12-5d56e15610d9-kube-api-access-6vsjw\") pod \"8cf49544-93d5-40fa-9f12-5d56e15610d9\" (UID: \"8cf49544-93d5-40fa-9f12-5d56e15610d9\") "
Oct 30 04:28:09.512585 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:09.512537    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-openshift-ca\") pod \"8cf49544-93d5-40fa-9f12-5d56e15610d9\" (UID: \"8cf49544-93d5-40fa-9f12-5d56e15610d9\") "
Oct 30 04:28:09.512969 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:09.512945    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-openshift-ca" pod "8cf49544-93d5-40fa-9f12-5d56e15610d9" (UID: "8cf49544-93d5-40fa-9f12-5d56e15610d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:28:09.513059 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:09.513030    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:28:09.514761 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:09.514743    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-xks-ca" pod "8cf49544-93d5-40fa-9f12-5d56e15610d9" (UID: "8cf49544-93d5-40fa-9f12-5d56e15610d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:28:09.515110 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:09.515095    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf49544-93d5-40fa-9f12-5d56e15610d9-kube-api-access-6vsjw" pod "8cf49544-93d5-40fa-9f12-5d56e15610d9" (UID: "8cf49544-93d5-40fa-9f12-5d56e15610d9"). InnerVolumeSpecName "kube-api-access-6vsjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:28:09.614264 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:09.614186    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf49544-93d5-40fa-9f12-5d56e15610d9-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:28:09.614264 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:09.614214    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vsjw\" (UniqueName: \"kubernetes.io/projected/8cf49544-93d5-40fa-9f12-5d56e15610d9-kube-api-access-6vsjw\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:28:10.355040 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:10.355012    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"
Oct 30 04:28:10.355350 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:10.355012    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt" event={"ID":"8cf49544-93d5-40fa-9f12-5d56e15610d9","Type":"ContainerDied","Data":"030dc0736f7cbe72ffe0402f61b29e1a836aece761b35ea7bf140e1e1b4259b0"}
Oct 30 04:28:10.355350 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:28:10.355316    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="030dc0736f7cbe72ffe0402f61b29e1a836aece761b35ea7bf140e1e1b4259b0"
Oct 30 04:28:11.751903 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:28:11.751862    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:28:41.751434 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:28:41.751393    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:29:02.480585 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.480544    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"]
Oct 30 04:29:02.481344 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.481322    5072 cpu_manager.go:414] "containerMap: removing container" podUID="8cf49544-93d5-40fa-9f12-5d56e15610d9" containerName="revoke-keys"
Oct 30 04:29:02.481474 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.481345    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf49544-93d5-40fa-9f12-5d56e15610d9" containerName="revoke-keys"
Oct 30 04:29:02.481562 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.481548    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf49544-93d5-40fa-9f12-5d56e15610d9" containerName="revoke-keys"
Oct 30 04:29:02.491574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.491551    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"
Oct 30 04:29:02.505823 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.505794    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"]
Oct 30 04:29:02.515270 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.515246    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3f5508\""
Oct 30 04:29:02.515436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.515405    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3f5508-dockercfg-4z28h\""
Oct 30 04:29:02.607865 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.607832    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"]
Oct 30 04:29:02.613621 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.613594    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"
Oct 30 04:29:02.620450 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.620421    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"]
Oct 30 04:29:02.649603 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.649565    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad831c01-3010-4d15-9213-3d3e14f73c49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv\" (UID: \"ad831c01-3010-4d15-9213-3d3e14f73c49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"
Oct 30 04:29:02.649748 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.649616    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66sd2\" (UniqueName: \"kubernetes.io/projected/ad831c01-3010-4d15-9213-3d3e14f73c49-kube-api-access-66sd2\") pod \"payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv\" (UID: \"ad831c01-3010-4d15-9213-3d3e14f73c49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"
Oct 30 04:29:02.750551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.750467    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb52ea29-0330-4a8a-b551-4425e7d39062-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l\" (UID: \"bb52ea29-0330-4a8a-b551-4425e7d39062\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"
Oct 30 04:29:02.750551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.750499    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwmw5\" (UniqueName: \"kubernetes.io/projected/bb52ea29-0330-4a8a-b551-4425e7d39062-kube-api-access-hwmw5\") pod \"payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l\" (UID: \"bb52ea29-0330-4a8a-b551-4425e7d39062\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"
Oct 30 04:29:02.750551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.750533    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad831c01-3010-4d15-9213-3d3e14f73c49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv\" (UID: \"ad831c01-3010-4d15-9213-3d3e14f73c49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"
Oct 30 04:29:02.750782 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.750672    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66sd2\" (UniqueName: \"kubernetes.io/projected/ad831c01-3010-4d15-9213-3d3e14f73c49-kube-api-access-66sd2\") pod \"payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv\" (UID: \"ad831c01-3010-4d15-9213-3d3e14f73c49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"
Oct 30 04:29:02.751250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.751229    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad831c01-3010-4d15-9213-3d3e14f73c49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv\" (UID: \"ad831c01-3010-4d15-9213-3d3e14f73c49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"
Oct 30 04:29:02.775182 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.775154    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66sd2\" (UniqueName: \"kubernetes.io/projected/ad831c01-3010-4d15-9213-3d3e14f73c49-kube-api-access-66sd2\") pod \"payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv\" (UID: \"ad831c01-3010-4d15-9213-3d3e14f73c49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"
Oct 30 04:29:02.806859 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.806832    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"
Oct 30 04:29:02.852172 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.852135    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb52ea29-0330-4a8a-b551-4425e7d39062-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l\" (UID: \"bb52ea29-0330-4a8a-b551-4425e7d39062\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"
Oct 30 04:29:02.852172 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.852176    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwmw5\" (UniqueName: \"kubernetes.io/projected/bb52ea29-0330-4a8a-b551-4425e7d39062-kube-api-access-hwmw5\") pod \"payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l\" (UID: \"bb52ea29-0330-4a8a-b551-4425e7d39062\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"
Oct 30 04:29:02.852844 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.852822    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb52ea29-0330-4a8a-b551-4425e7d39062-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l\" (UID: \"bb52ea29-0330-4a8a-b551-4425e7d39062\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"
Oct 30 04:29:02.877652 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.877624    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwmw5\" (UniqueName: \"kubernetes.io/projected/bb52ea29-0330-4a8a-b551-4425e7d39062-kube-api-access-hwmw5\") pod \"payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l\" (UID: \"bb52ea29-0330-4a8a-b551-4425e7d39062\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"
Oct 30 04:29:02.931340 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:02.931311    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"
Oct 30 04:29:03.108925 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:03.108895    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"]
Oct 30 04:29:03.110048 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:29:03.110020    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad831c01_3010_4d15_9213_3d3e14f73c49.slice/crio-e9897fec2f6448c377b1d1698abfe7ea746f7f10862dcfff6275c33e61cbea75 WatchSource:0}: Error finding container e9897fec2f6448c377b1d1698abfe7ea746f7f10862dcfff6275c33e61cbea75: Status 404 returned error can't find the container with id e9897fec2f6448c377b1d1698abfe7ea746f7f10862dcfff6275c33e61cbea75
Oct 30 04:29:03.235339 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:29:03.235308    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb52ea29_0330_4a8a_b551_4425e7d39062.slice/crio-8fda99c523d8108cbda2c727fdba0f95beb3eb44c4f4febea3e8757cb67133b5 WatchSource:0}: Error finding container 8fda99c523d8108cbda2c727fdba0f95beb3eb44c4f4febea3e8757cb67133b5: Status 404 returned error can't find the container with id 8fda99c523d8108cbda2c727fdba0f95beb3eb44c4f4febea3e8757cb67133b5
Oct 30 04:29:03.236347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:03.236329    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"]
Oct 30 04:29:03.726289 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:03.726247    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l" event={"ID":"bb52ea29-0330-4a8a-b551-4425e7d39062","Type":"ContainerStarted","Data":"8fda99c523d8108cbda2c727fdba0f95beb3eb44c4f4febea3e8757cb67133b5"}
Oct 30 04:29:03.729188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:03.729154    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv" event={"ID":"ad831c01-3010-4d15-9213-3d3e14f73c49","Type":"ContainerStarted","Data":"e9897fec2f6448c377b1d1698abfe7ea746f7f10862dcfff6275c33e61cbea75"}
Oct 30 04:29:05.745290 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:05.745252    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l" event={"ID":"bb52ea29-0330-4a8a-b551-4425e7d39062","Type":"ContainerStarted","Data":"17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f"}
Oct 30 04:29:05.745775 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:05.745434    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"
Oct 30 04:29:05.746992 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:05.746967    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv" event={"ID":"ad831c01-3010-4d15-9213-3d3e14f73c49","Type":"ContainerStarted","Data":"82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2"}
Oct 30 04:29:05.747133 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:05.747038    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"
Oct 30 04:29:05.791773 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:05.791720    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l" podStartSLOduration=1.5331151360000002 podStartE2EDuration="3.79170601s" totalImagesPullingTime="2.258590874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:05.790941394 +0000 UTC m=+1925.532364502" watchObservedRunningTime="2026-10-30 04:29:05.79170601 +0000 UTC m=+1925.533129147"
Oct 30 04:29:05.834378 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:05.834330    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv" podStartSLOduration=1.366731747 podStartE2EDuration="3.834315317s" totalImagesPullingTime="2.46758357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:05.832455718 +0000 UTC m=+1925.573878827" watchObservedRunningTime="2026-10-30 04:29:05.834315317 +0000 UTC m=+1925.575738424"
Oct 30 04:29:11.751011 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:29:11.750937    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:29:14.156915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.156878    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"]
Oct 30 04:29:14.157585 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.157184    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4" podUID="5fd1e079-8a0d-4aac-98a1-33347d2d515a" containerName="payload-pre-processing" containerID="cri-o://16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393" gracePeriod=30
Oct 30 04:29:14.480142 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.479995    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"
Oct 30 04:29:14.596447 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.596418    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd1e079-8a0d-4aac-98a1-33347d2d515a-kube-api-access-dhz2b\" (UniqueName: \"kubernetes.io/projected/5fd1e079-8a0d-4aac-98a1-33347d2d515a-kube-api-access-dhz2b\") pod \"5fd1e079-8a0d-4aac-98a1-33347d2d515a\" (UID: \"5fd1e079-8a0d-4aac-98a1-33347d2d515a\") "
Oct 30 04:29:14.596651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.596521    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fd1e079-8a0d-4aac-98a1-33347d2d515a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd1e079-8a0d-4aac-98a1-33347d2d515a-plugins-config-volume\") pod \"5fd1e079-8a0d-4aac-98a1-33347d2d515a\" (UID: \"5fd1e079-8a0d-4aac-98a1-33347d2d515a\") "
Oct 30 04:29:14.596995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.596959    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd1e079-8a0d-4aac-98a1-33347d2d515a-plugins-config-volume" pod "5fd1e079-8a0d-4aac-98a1-33347d2d515a" (UID: "5fd1e079-8a0d-4aac-98a1-33347d2d515a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:14.599229 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.599201    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd1e079-8a0d-4aac-98a1-33347d2d515a-kube-api-access-dhz2b" pod "5fd1e079-8a0d-4aac-98a1-33347d2d515a" (UID: "5fd1e079-8a0d-4aac-98a1-33347d2d515a"). InnerVolumeSpecName "kube-api-access-dhz2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:14.698462 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.698434    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhz2b\" (UniqueName: \"kubernetes.io/projected/5fd1e079-8a0d-4aac-98a1-33347d2d515a-kube-api-access-dhz2b\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:29:14.698462 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.698460    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd1e079-8a0d-4aac-98a1-33347d2d515a-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:29:14.803888 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.803860    5072 generic.go:356] "Generic (PLEG): container finished" podID="5fd1e079-8a0d-4aac-98a1-33347d2d515a" containerID="16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393" exitCode=0
Oct 30 04:29:14.804026 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.803908    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"
Oct 30 04:29:14.804026 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.803946    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4" event={"ID":"5fd1e079-8a0d-4aac-98a1-33347d2d515a","Type":"ContainerDied","Data":"16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393"}
Oct 30 04:29:14.804026 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.804000    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4" event={"ID":"5fd1e079-8a0d-4aac-98a1-33347d2d515a","Type":"ContainerDied","Data":"b21a1052ac3d05fd9c758efc936a858a7a2bdeed32ba7754b6189ede5851678a"}
Oct 30 04:29:14.804026 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.804024    5072 scope.go:122] "RemoveContainer" containerID="16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393"
Oct 30 04:29:14.814184 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.814166    5072 scope.go:122] "RemoveContainer" containerID="16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393"
Oct 30 04:29:14.814484 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:29:14.814465    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393\": container with ID starting with 16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393 not found: ID does not exist" containerID="16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393"
Oct 30 04:29:14.814553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.814495    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393"} err="failed to get container status \"16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393\": rpc error: code = NotFound desc = could not find container \"16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393\": container with ID starting with 16e426c7b4998d64ed8d97ee87a4a0ba90ab4ecaffdbf28b4ae2bdad596db393 not found: ID does not exist"
Oct 30 04:29:14.858431 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.858404    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"]
Oct 30 04:29:14.872704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:14.872666    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-57c5db57f5-v9xw4"]
Oct 30 04:29:16.407827 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:16.407793    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd1e079-8a0d-4aac-98a1-33347d2d515a" path="/var/lib/kubelet/pods/5fd1e079-8a0d-4aac-98a1-33347d2d515a/volumes"
Oct 30 04:29:16.759184 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:16.759151    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"
Oct 30 04:29:16.759331 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:16.759222    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"
Oct 30 04:29:26.928465 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:26.928428    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"]
Oct 30 04:29:26.929190 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:26.929126    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm" podUID="64b1201f-aea5-4193-84fd-a9c8a1b2779d" containerName="payload-pre-processing" containerID="cri-o://085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb" gracePeriod=30
Oct 30 04:29:27.237100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.236863    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"
Oct 30 04:29:27.352734 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.352700    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b1201f-aea5-4193-84fd-a9c8a1b2779d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b1201f-aea5-4193-84fd-a9c8a1b2779d-plugins-config-volume\") pod \"64b1201f-aea5-4193-84fd-a9c8a1b2779d\" (UID: \"64b1201f-aea5-4193-84fd-a9c8a1b2779d\") "
Oct 30 04:29:27.352917 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.352887    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b1201f-aea5-4193-84fd-a9c8a1b2779d-kube-api-access-rfxt2\" (UniqueName: \"kubernetes.io/projected/64b1201f-aea5-4193-84fd-a9c8a1b2779d-kube-api-access-rfxt2\") pod \"64b1201f-aea5-4193-84fd-a9c8a1b2779d\" (UID: \"64b1201f-aea5-4193-84fd-a9c8a1b2779d\") "
Oct 30 04:29:27.353213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.353185    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b1201f-aea5-4193-84fd-a9c8a1b2779d-plugins-config-volume" pod "64b1201f-aea5-4193-84fd-a9c8a1b2779d" (UID: "64b1201f-aea5-4193-84fd-a9c8a1b2779d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:27.353389 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.353368    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b1201f-aea5-4193-84fd-a9c8a1b2779d-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:29:27.355143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.355071    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b1201f-aea5-4193-84fd-a9c8a1b2779d-kube-api-access-rfxt2" pod "64b1201f-aea5-4193-84fd-a9c8a1b2779d" (UID: "64b1201f-aea5-4193-84fd-a9c8a1b2779d"). InnerVolumeSpecName "kube-api-access-rfxt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:27.454427 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.454367    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfxt2\" (UniqueName: \"kubernetes.io/projected/64b1201f-aea5-4193-84fd-a9c8a1b2779d-kube-api-access-rfxt2\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:29:27.895665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.895583    5072 generic.go:356] "Generic (PLEG): container finished" podID="64b1201f-aea5-4193-84fd-a9c8a1b2779d" containerID="085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb" exitCode=0
Oct 30 04:29:27.895665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.895628    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"
Oct 30 04:29:27.895844 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.895672    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm" event={"ID":"64b1201f-aea5-4193-84fd-a9c8a1b2779d","Type":"ContainerDied","Data":"085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb"}
Oct 30 04:29:27.895844 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.895729    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm" event={"ID":"64b1201f-aea5-4193-84fd-a9c8a1b2779d","Type":"ContainerDied","Data":"a72cb0318d0cf8fabc98a7e1f787e54f0b5a93719872af8e5534439ab262f635"}
Oct 30 04:29:27.895844 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.895758    5072 scope.go:122] "RemoveContainer" containerID="085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb"
Oct 30 04:29:27.907647 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.907628    5072 scope.go:122] "RemoveContainer" containerID="085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb"
Oct 30 04:29:27.907897 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:29:27.907875    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb\": container with ID starting with 085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb not found: ID does not exist" containerID="085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb"
Oct 30 04:29:27.907938 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.907910    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb"} err="failed to get container status \"085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb\": rpc error: code = NotFound desc = could not find container \"085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb\": container with ID starting with 085b4ad880dd46122130196655e2fb148bb1a85a2c29c19c9c4e1285d3a6facb not found: ID does not exist"
Oct 30 04:29:27.953677 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.953649    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"]
Oct 30 04:29:27.966356 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:27.966330    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7dd5c45647-m5lzm"]
Oct 30 04:29:28.408685 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:28.408655    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b1201f-aea5-4193-84fd-a9c8a1b2779d" path="/var/lib/kubelet/pods/64b1201f-aea5-4193-84fd-a9c8a1b2779d/volumes"
Oct 30 04:29:33.287348 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.287301    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"]
Oct 30 04:29:33.287861 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.287579    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9" podUID="d2ecbf17-3341-4e24-9aab-b88cb1bd10e9" containerName="payload-pre-processing" containerID="cri-o://9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c" gracePeriod=30
Oct 30 04:29:33.607136 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.607103    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"
Oct 30 04:29:33.630186 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.627684    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-plugins-config-volume\") pod \"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9\" (UID: \"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9\") "
Oct 30 04:29:33.630186 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.627729    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-kube-api-access-z6rqd\" (UniqueName: \"kubernetes.io/projected/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-kube-api-access-z6rqd\") pod \"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9\" (UID: \"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9\") "
Oct 30 04:29:33.631058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.631012    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-plugins-config-volume" pod "d2ecbf17-3341-4e24-9aab-b88cb1bd10e9" (UID: "d2ecbf17-3341-4e24-9aab-b88cb1bd10e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:33.649267 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.649198    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-kube-api-access-z6rqd" pod "d2ecbf17-3341-4e24-9aab-b88cb1bd10e9" (UID: "d2ecbf17-3341-4e24-9aab-b88cb1bd10e9"). InnerVolumeSpecName "kube-api-access-z6rqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:33.729458 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.729425    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:29:33.729458 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.729458    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6rqd\" (UniqueName: \"kubernetes.io/projected/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9-kube-api-access-z6rqd\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:29:33.941212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.941183    5072 generic.go:356] "Generic (PLEG): container finished" podID="d2ecbf17-3341-4e24-9aab-b88cb1bd10e9" containerID="9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c" exitCode=0
Oct 30 04:29:33.941368 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.941234    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"
Oct 30 04:29:33.941368 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.941237    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9" event={"ID":"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9","Type":"ContainerDied","Data":"9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c"}
Oct 30 04:29:33.941368 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.941351    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9" event={"ID":"d2ecbf17-3341-4e24-9aab-b88cb1bd10e9","Type":"ContainerDied","Data":"ab5227831b0ee1b47bb39ae19c3631fe93feedb99eb45e0ad9cd5b87e7603093"}
Oct 30 04:29:33.941467 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.941373    5072 scope.go:122] "RemoveContainer" containerID="9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c"
Oct 30 04:29:33.951722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.951705    5072 scope.go:122] "RemoveContainer" containerID="9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c"
Oct 30 04:29:33.952005 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:29:33.951983    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c\": container with ID starting with 9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c not found: ID does not exist" containerID="9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c"
Oct 30 04:29:33.952055 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.952014    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c"} err="failed to get container status \"9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c\": rpc error: code = NotFound desc = could not find container \"9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c\": container with ID starting with 9b096637f9f89dd52796f83b2505337740f0543dfea85d6609977ed78350897c not found: ID does not exist"
Oct 30 04:29:33.995336 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:33.995315    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"]
Oct 30 04:29:34.008036 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:34.008013    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-66457b6895-g8dj9"]
Oct 30 04:29:34.411726 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:34.411647    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ecbf17-3341-4e24-9aab-b88cb1bd10e9" path="/var/lib/kubelet/pods/d2ecbf17-3341-4e24-9aab-b88cb1bd10e9/volumes"
Oct 30 04:29:41.769398 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:29:41.769292    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:29:51.615314 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:51.615278    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"]
Oct 30 04:29:51.644163 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:51.644129    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"]
Oct 30 04:29:56.616179 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:56.616071    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l" podUID="bb52ea29-0330-4a8a-b551-4425e7d39062" containerName="payload-processing" containerID="cri-o://17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f" gracePeriod=25
Oct 30 04:29:56.644988 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:56.644933    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv" podUID="ad831c01-3010-4d15-9213-3d3e14f73c49" containerName="payload-pre-processing" containerID="cri-o://82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2" gracePeriod=25
Oct 30 04:29:56.757037 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:29:56.757012    5072 logging.go:55] [core] [Channel #51 SubChannel #53]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.87:50052", ServerName: "10.132.0.87:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.87:50052: connect: connection refused"
Oct 30 04:29:56.757037 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:29:56.757025    5072 logging.go:55] [core] [Channel #50 SubChannel #52]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.86:50052", ServerName: "10.132.0.86:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.86:50052: connect: connection refused"
Oct 30 04:29:56.934438 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:56.934414    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"
Oct 30 04:29:56.972326 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:56.972302    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"
Oct 30 04:29:57.109779 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.109748    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb52ea29-0330-4a8a-b551-4425e7d39062-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb52ea29-0330-4a8a-b551-4425e7d39062-plugins-config-volume\") pod \"bb52ea29-0330-4a8a-b551-4425e7d39062\" (UID: \"bb52ea29-0330-4a8a-b551-4425e7d39062\") "
Oct 30 04:29:57.110013 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.109800    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb52ea29-0330-4a8a-b551-4425e7d39062-kube-api-access-hwmw5\" (UniqueName: \"kubernetes.io/projected/bb52ea29-0330-4a8a-b551-4425e7d39062-kube-api-access-hwmw5\") pod \"bb52ea29-0330-4a8a-b551-4425e7d39062\" (UID: \"bb52ea29-0330-4a8a-b551-4425e7d39062\") "
Oct 30 04:29:57.110013 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.109924    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad831c01-3010-4d15-9213-3d3e14f73c49-kube-api-access-66sd2\" (UniqueName: \"kubernetes.io/projected/ad831c01-3010-4d15-9213-3d3e14f73c49-kube-api-access-66sd2\") pod \"ad831c01-3010-4d15-9213-3d3e14f73c49\" (UID: \"ad831c01-3010-4d15-9213-3d3e14f73c49\") "
Oct 30 04:29:57.110179 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.110036    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad831c01-3010-4d15-9213-3d3e14f73c49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad831c01-3010-4d15-9213-3d3e14f73c49-plugins-config-volume\") pod \"ad831c01-3010-4d15-9213-3d3e14f73c49\" (UID: \"ad831c01-3010-4d15-9213-3d3e14f73c49\") "
Oct 30 04:29:57.110238 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.110214    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb52ea29-0330-4a8a-b551-4425e7d39062-plugins-config-volume" pod "bb52ea29-0330-4a8a-b551-4425e7d39062" (UID: "bb52ea29-0330-4a8a-b551-4425e7d39062"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:57.110563 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.110539    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb52ea29-0330-4a8a-b551-4425e7d39062-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:29:57.110833 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.110809    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad831c01-3010-4d15-9213-3d3e14f73c49-plugins-config-volume" pod "ad831c01-3010-4d15-9213-3d3e14f73c49" (UID: "ad831c01-3010-4d15-9213-3d3e14f73c49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:57.112267 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.112244    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad831c01-3010-4d15-9213-3d3e14f73c49-kube-api-access-66sd2" pod "ad831c01-3010-4d15-9213-3d3e14f73c49" (UID: "ad831c01-3010-4d15-9213-3d3e14f73c49"). InnerVolumeSpecName "kube-api-access-66sd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:57.112267 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.112257    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb52ea29-0330-4a8a-b551-4425e7d39062-kube-api-access-hwmw5" pod "bb52ea29-0330-4a8a-b551-4425e7d39062" (UID: "bb52ea29-0330-4a8a-b551-4425e7d39062"). InnerVolumeSpecName "kube-api-access-hwmw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:57.125370 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.125343    5072 generic.go:356] "Generic (PLEG): container finished" podID="ad831c01-3010-4d15-9213-3d3e14f73c49" containerID="82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2" exitCode=0
Oct 30 04:29:57.125466 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.125391    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"
Oct 30 04:29:57.125466 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.125429    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv" event={"ID":"ad831c01-3010-4d15-9213-3d3e14f73c49","Type":"ContainerDied","Data":"82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2"}
Oct 30 04:29:57.125554 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.125483    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv" event={"ID":"ad831c01-3010-4d15-9213-3d3e14f73c49","Type":"ContainerDied","Data":"e9897fec2f6448c377b1d1698abfe7ea746f7f10862dcfff6275c33e61cbea75"}
Oct 30 04:29:57.125554 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.125512    5072 scope.go:122] "RemoveContainer" containerID="82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2"
Oct 30 04:29:57.127262 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.127239    5072 generic.go:356] "Generic (PLEG): container finished" podID="bb52ea29-0330-4a8a-b551-4425e7d39062" containerID="17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f" exitCode=0
Oct 30 04:29:57.127366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.127305    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l" event={"ID":"bb52ea29-0330-4a8a-b551-4425e7d39062","Type":"ContainerDied","Data":"17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f"}
Oct 30 04:29:57.127366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.127346    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l" event={"ID":"bb52ea29-0330-4a8a-b551-4425e7d39062","Type":"ContainerDied","Data":"8fda99c523d8108cbda2c727fdba0f95beb3eb44c4f4febea3e8757cb67133b5"}
Oct 30 04:29:57.127486 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.127412    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"
Oct 30 04:29:57.140021 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.139999    5072 scope.go:122] "RemoveContainer" containerID="82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2"
Oct 30 04:29:57.140506 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:29:57.140475    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2\": container with ID starting with 82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2 not found: ID does not exist" containerID="82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2"
Oct 30 04:29:57.140616 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.140515    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2"} err="failed to get container status \"82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2\": rpc error: code = NotFound desc = could not find container \"82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2\": container with ID starting with 82ca2bec080d60a128f6fb0ba33233ffa67c5d592cc9cded9bb7f9c0cacd25e2 not found: ID does not exist"
Oct 30 04:29:57.140616 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.140542    5072 scope.go:122] "RemoveContainer" containerID="17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f"
Oct 30 04:29:57.151871 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.151853    5072 scope.go:122] "RemoveContainer" containerID="17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f"
Oct 30 04:29:57.152159 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:29:57.152136    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f\": container with ID starting with 17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f not found: ID does not exist" containerID="17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f"
Oct 30 04:29:57.152215 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.152166    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f"} err="failed to get container status \"17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f\": rpc error: code = NotFound desc = could not find container \"17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f\": container with ID starting with 17e77f4cb915909977c93c65e07998e02800eed87eae4b1ff4135aa0d5a10c3f not found: ID does not exist"
Oct 30 04:29:57.178882 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.178857    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"]
Oct 30 04:29:57.183041 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.183017    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv"]
Oct 30 04:29:57.212170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.212136    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad831c01-3010-4d15-9213-3d3e14f73c49-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:29:57.212170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.212169    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwmw5\" (UniqueName: \"kubernetes.io/projected/bb52ea29-0330-4a8a-b551-4425e7d39062-kube-api-access-hwmw5\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:29:57.212403 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.212186    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66sd2\" (UniqueName: \"kubernetes.io/projected/ad831c01-3010-4d15-9213-3d3e14f73c49-kube-api-access-66sd2\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:29:57.218394 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.218329    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"]
Oct 30 04:29:57.221268 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.221243    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l"]
Oct 30 04:29:57.757616 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.757566    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-9d649549f-kmwcv" podUID="ad831c01-3010-4d15-9213-3d3e14f73c49" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.86:50052\" within 1s: context deadline exceeded"
Oct 30 04:29:57.758040 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:57.757625    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-6c8d9cbd78-rgj9l" podUID="bb52ea29-0330-4a8a-b551-4425e7d39062" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.87:50052\" within 1s: context deadline exceeded"
Oct 30 04:29:57.758040 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:29:57.757758    5072 logging.go:55] [core] [Channel #51 SubChannel #53]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.87:50052", ServerName: "10.132.0.87:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.87:50052: operation was canceled"
Oct 30 04:29:58.467130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:58.452361    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad831c01-3010-4d15-9213-3d3e14f73c49" path="/var/lib/kubelet/pods/ad831c01-3010-4d15-9213-3d3e14f73c49/volumes"
Oct 30 04:29:58.467130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:29:58.453647    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb52ea29-0330-4a8a-b551-4425e7d39062" path="/var/lib/kubelet/pods/bb52ea29-0330-4a8a-b551-4425e7d39062/volumes"
Oct 30 04:30:00.150029 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.149996    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"]
Oct 30 04:30:00.150542 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150528    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ad831c01-3010-4d15-9213-3d3e14f73c49" containerName="payload-pre-processing"
Oct 30 04:30:00.150590 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150543    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad831c01-3010-4d15-9213-3d3e14f73c49" containerName="payload-pre-processing"
Oct 30 04:30:00.150590 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150561    5072 cpu_manager.go:414] "containerMap: removing container" podUID="bb52ea29-0330-4a8a-b551-4425e7d39062" containerName="payload-processing"
Oct 30 04:30:00.150590 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150567    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb52ea29-0330-4a8a-b551-4425e7d39062" containerName="payload-processing"
Oct 30 04:30:00.150590 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150581    5072 cpu_manager.go:414] "containerMap: removing container" podUID="d2ecbf17-3341-4e24-9aab-b88cb1bd10e9" containerName="payload-pre-processing"
Oct 30 04:30:00.150590 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150586    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ecbf17-3341-4e24-9aab-b88cb1bd10e9" containerName="payload-pre-processing"
Oct 30 04:30:00.150735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150596    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5fd1e079-8a0d-4aac-98a1-33347d2d515a" containerName="payload-pre-processing"
Oct 30 04:30:00.150735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150601    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd1e079-8a0d-4aac-98a1-33347d2d515a" containerName="payload-pre-processing"
Oct 30 04:30:00.150735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150623    5072 cpu_manager.go:414] "containerMap: removing container" podUID="64b1201f-aea5-4193-84fd-a9c8a1b2779d" containerName="payload-pre-processing"
Oct 30 04:30:00.150735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150628    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b1201f-aea5-4193-84fd-a9c8a1b2779d" containerName="payload-pre-processing"
Oct 30 04:30:00.150735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150702    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad831c01-3010-4d15-9213-3d3e14f73c49" containerName="payload-pre-processing"
Oct 30 04:30:00.150735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150712    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb52ea29-0330-4a8a-b551-4425e7d39062" containerName="payload-processing"
Oct 30 04:30:00.150735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150720    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ecbf17-3341-4e24-9aab-b88cb1bd10e9" containerName="payload-pre-processing"
Oct 30 04:30:00.150735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150726    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd1e079-8a0d-4aac-98a1-33347d2d515a" containerName="payload-pre-processing"
Oct 30 04:30:00.150735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.150735    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b1201f-aea5-4193-84fd-a9c8a1b2779d" containerName="payload-pre-processing"
Oct 30 04:30:00.155596 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.155576    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"
Oct 30 04:30:00.171373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.171351    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"]
Oct 30 04:30:00.175393 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.175370    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"
Oct 30 04:30:00.179020 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.178999    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"]
Oct 30 04:30:00.181521 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.181500    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:30:00.194214 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.193547    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"]
Oct 30 04:30:00.198697 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.198675    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"]
Oct 30 04:30:00.198831 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.198787    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"
Oct 30 04:30:00.206306 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.206284    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"]
Oct 30 04:30:00.210407 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.210378    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"
Oct 30 04:30:00.217808 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.217782    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"]
Oct 30 04:30:00.238055 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.238030    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"]
Oct 30 04:30:00.269292 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.269265    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657b39ad-f9e4-4635-a144-3f434f7a193e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465\" (UID: \"657b39ad-f9e4-4635-a144-3f434f7a193e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"
Oct 30 04:30:00.269448 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.269321    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df1e95a-34b9-45ac-beb6-523c9de84f05-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888910-qwp8x\" (UID: \"5df1e95a-34b9-45ac-beb6-523c9de84f05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"
Oct 30 04:30:00.269448 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.269407    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2vx8\" (UniqueName: \"kubernetes.io/projected/5df1e95a-34b9-45ac-beb6-523c9de84f05-kube-api-access-j2vx8\") pod \"maas-api-key-cleanup-29888910-qwp8x\" (UID: \"5df1e95a-34b9-45ac-beb6-523c9de84f05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"
Oct 30 04:30:00.269554 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.269445    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4sc7\" (UniqueName: \"kubernetes.io/projected/657b39ad-f9e4-4635-a144-3f434f7a193e-kube-api-access-k4sc7\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465\" (UID: \"657b39ad-f9e4-4635-a144-3f434f7a193e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"
Oct 30 04:30:00.269554 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.269495    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafe1f28-6810-4682-84fa-a31518286947-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz\" (UID: \"cafe1f28-6810-4682-84fa-a31518286947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"
Oct 30 04:30:00.269662 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.269649    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5snw4\" (UniqueName: \"kubernetes.io/projected/cafe1f28-6810-4682-84fa-a31518286947-kube-api-access-5snw4\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz\" (UID: \"cafe1f28-6810-4682-84fa-a31518286947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"
Oct 30 04:30:00.269700 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.269677    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd106df6-76b0-4919-8ec1-3f34d9da3d02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs\" (UID: \"dd106df6-76b0-4919-8ec1-3f34d9da3d02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"
Oct 30 04:30:00.269799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.269699    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk9wz\" (UniqueName: \"kubernetes.io/projected/dd106df6-76b0-4919-8ec1-3f34d9da3d02-kube-api-access-zk9wz\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs\" (UID: \"dd106df6-76b0-4919-8ec1-3f34d9da3d02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"
Oct 30 04:30:00.370813 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.370780    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4sc7\" (UniqueName: \"kubernetes.io/projected/657b39ad-f9e4-4635-a144-3f434f7a193e-kube-api-access-k4sc7\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465\" (UID: \"657b39ad-f9e4-4635-a144-3f434f7a193e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"
Oct 30 04:30:00.370813 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.370817    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafe1f28-6810-4682-84fa-a31518286947-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz\" (UID: \"cafe1f28-6810-4682-84fa-a31518286947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"
Oct 30 04:30:00.371014 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.370969    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5snw4\" (UniqueName: \"kubernetes.io/projected/cafe1f28-6810-4682-84fa-a31518286947-kube-api-access-5snw4\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz\" (UID: \"cafe1f28-6810-4682-84fa-a31518286947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"
Oct 30 04:30:00.371014 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.371003    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd106df6-76b0-4919-8ec1-3f34d9da3d02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs\" (UID: \"dd106df6-76b0-4919-8ec1-3f34d9da3d02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"
Oct 30 04:30:00.371173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.371038    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk9wz\" (UniqueName: \"kubernetes.io/projected/dd106df6-76b0-4919-8ec1-3f34d9da3d02-kube-api-access-zk9wz\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs\" (UID: \"dd106df6-76b0-4919-8ec1-3f34d9da3d02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"
Oct 30 04:30:00.371173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.371126    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657b39ad-f9e4-4635-a144-3f434f7a193e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465\" (UID: \"657b39ad-f9e4-4635-a144-3f434f7a193e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"
Oct 30 04:30:00.371173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.371169    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df1e95a-34b9-45ac-beb6-523c9de84f05-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888910-qwp8x\" (UID: \"5df1e95a-34b9-45ac-beb6-523c9de84f05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"
Oct 30 04:30:00.371330 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.371190    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2vx8\" (UniqueName: \"kubernetes.io/projected/5df1e95a-34b9-45ac-beb6-523c9de84f05-kube-api-access-j2vx8\") pod \"maas-api-key-cleanup-29888910-qwp8x\" (UID: \"5df1e95a-34b9-45ac-beb6-523c9de84f05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"
Oct 30 04:30:00.371623 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.371605    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafe1f28-6810-4682-84fa-a31518286947-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz\" (UID: \"cafe1f28-6810-4682-84fa-a31518286947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"
Oct 30 04:30:00.371778 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.371761    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657b39ad-f9e4-4635-a144-3f434f7a193e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465\" (UID: \"657b39ad-f9e4-4635-a144-3f434f7a193e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"
Oct 30 04:30:00.371814 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.371793    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd106df6-76b0-4919-8ec1-3f34d9da3d02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs\" (UID: \"dd106df6-76b0-4919-8ec1-3f34d9da3d02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"
Oct 30 04:30:00.371885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.371868    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df1e95a-34b9-45ac-beb6-523c9de84f05-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888910-qwp8x\" (UID: \"5df1e95a-34b9-45ac-beb6-523c9de84f05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"
Oct 30 04:30:00.395368 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.395345    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5snw4\" (UniqueName: \"kubernetes.io/projected/cafe1f28-6810-4682-84fa-a31518286947-kube-api-access-5snw4\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz\" (UID: \"cafe1f28-6810-4682-84fa-a31518286947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"
Oct 30 04:30:00.395509 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.395419    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4sc7\" (UniqueName: \"kubernetes.io/projected/657b39ad-f9e4-4635-a144-3f434f7a193e-kube-api-access-k4sc7\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465\" (UID: \"657b39ad-f9e4-4635-a144-3f434f7a193e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"
Oct 30 04:30:00.395571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.395558    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2vx8\" (UniqueName: \"kubernetes.io/projected/5df1e95a-34b9-45ac-beb6-523c9de84f05-kube-api-access-j2vx8\") pod \"maas-api-key-cleanup-29888910-qwp8x\" (UID: \"5df1e95a-34b9-45ac-beb6-523c9de84f05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"
Oct 30 04:30:00.395943 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.395922    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk9wz\" (UniqueName: \"kubernetes.io/projected/dd106df6-76b0-4919-8ec1-3f34d9da3d02-kube-api-access-zk9wz\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs\" (UID: \"dd106df6-76b0-4919-8ec1-3f34d9da3d02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"
Oct 30 04:30:00.483663 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.483627    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:30:00.489191 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.489170    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"
Oct 30 04:30:00.493245 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.493225    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"
Oct 30 04:30:00.522215 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.522139    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"
Oct 30 04:30:00.529578 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.529552    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"
Oct 30 04:30:00.811713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.811689    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"]
Oct 30 04:30:00.815383 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:30:00.815357    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafe1f28_6810_4682_84fa_a31518286947.slice/crio-15c38255416d09beb0ec18d9e4c3ddc94fab2f29c4dbb567954837ad1cc35519 WatchSource:0}: Error finding container 15c38255416d09beb0ec18d9e4c3ddc94fab2f29c4dbb567954837ad1cc35519: Status 404 returned error can't find the container with id 15c38255416d09beb0ec18d9e4c3ddc94fab2f29c4dbb567954837ad1cc35519
Oct 30 04:30:00.822466 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.822442    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"]
Oct 30 04:30:00.827419 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:30:00.827398    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd106df6_76b0_4919_8ec1_3f34d9da3d02.slice/crio-bd531e24d6f90c087f2a3e3e0fac61219b5b76b42707dfad423d339efec29b81 WatchSource:0}: Error finding container bd531e24d6f90c087f2a3e3e0fac61219b5b76b42707dfad423d339efec29b81: Status 404 returned error can't find the container with id bd531e24d6f90c087f2a3e3e0fac61219b5b76b42707dfad423d339efec29b81
Oct 30 04:30:00.865633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.865606    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"]
Oct 30 04:30:00.867347 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:30:00.867324    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df1e95a_34b9_45ac_beb6_523c9de84f05.slice/crio-037d5ce6138368dbbff7e9f96e38fa2fc1261af3acf4721f8e2b00d6a7a81f05 WatchSource:0}: Error finding container 037d5ce6138368dbbff7e9f96e38fa2fc1261af3acf4721f8e2b00d6a7a81f05: Status 404 returned error can't find the container with id 037d5ce6138368dbbff7e9f96e38fa2fc1261af3acf4721f8e2b00d6a7a81f05
Oct 30 04:30:00.867758 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:30:00.867738    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657b39ad_f9e4_4635_a144_3f434f7a193e.slice/crio-d9feaf5e97c8cd4b6d9414ef6766325c70884de72418d88638b92ad36946a3c6 WatchSource:0}: Error finding container d9feaf5e97c8cd4b6d9414ef6766325c70884de72418d88638b92ad36946a3c6: Status 404 returned error can't find the container with id d9feaf5e97c8cd4b6d9414ef6766325c70884de72418d88638b92ad36946a3c6
Oct 30 04:30:00.869000 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:00.868975    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"]
Oct 30 04:30:01.155862 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:01.155822    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x" event={"ID":"5df1e95a-34b9-45ac-beb6-523c9de84f05","Type":"ContainerStarted","Data":"037d5ce6138368dbbff7e9f96e38fa2fc1261af3acf4721f8e2b00d6a7a81f05"}
Oct 30 04:30:01.157545 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:01.157511    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs" event={"ID":"dd106df6-76b0-4919-8ec1-3f34d9da3d02","Type":"ContainerStarted","Data":"bd531e24d6f90c087f2a3e3e0fac61219b5b76b42707dfad423d339efec29b81"}
Oct 30 04:30:01.159050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:01.159022    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz" event={"ID":"cafe1f28-6810-4682-84fa-a31518286947","Type":"ContainerStarted","Data":"15c38255416d09beb0ec18d9e4c3ddc94fab2f29c4dbb567954837ad1cc35519"}
Oct 30 04:30:01.160516 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:01.160495    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465" event={"ID":"657b39ad-f9e4-4635-a144-3f434f7a193e","Type":"ContainerStarted","Data":"d9feaf5e97c8cd4b6d9414ef6766325c70884de72418d88638b92ad36946a3c6"}
Oct 30 04:30:02.168956 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:02.168881    5072 generic.go:356] "Generic (PLEG): container finished" podID="657b39ad-f9e4-4635-a144-3f434f7a193e" containerID="33a3db1481991849ff47c637ad3c7943e226b15fde8d09cadc908f9cc3420e0b" exitCode=60
Oct 30 04:30:02.169424 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:02.168973    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465" event={"ID":"657b39ad-f9e4-4635-a144-3f434f7a193e","Type":"ContainerDied","Data":"33a3db1481991849ff47c637ad3c7943e226b15fde8d09cadc908f9cc3420e0b"}
Oct 30 04:30:02.169424 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:02.169266    5072 scope.go:122] "RemoveContainer" containerID="33a3db1481991849ff47c637ad3c7943e226b15fde8d09cadc908f9cc3420e0b"
Oct 30 04:30:02.170720 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:02.170699    5072 generic.go:356] "Generic (PLEG): container finished" podID="5df1e95a-34b9-45ac-beb6-523c9de84f05" containerID="8272c95f774c72de8927bd6f6679201420f9dcbe91d5693c4bcae8fd20a12a84" exitCode=60
Oct 30 04:30:02.170842 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:02.170761    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x" event={"ID":"5df1e95a-34b9-45ac-beb6-523c9de84f05","Type":"ContainerDied","Data":"8272c95f774c72de8927bd6f6679201420f9dcbe91d5693c4bcae8fd20a12a84"}
Oct 30 04:30:02.171063 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:02.171042    5072 scope.go:122] "RemoveContainer" containerID="8272c95f774c72de8927bd6f6679201420f9dcbe91d5693c4bcae8fd20a12a84"
Oct 30 04:30:02.172406 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:02.172384    5072 generic.go:356] "Generic (PLEG): container finished" podID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" containerID="aaeb33152409eddbce6e747d80edd34e4f41974e0fbc5640a33a27258e9a8f66" exitCode=60
Oct 30 04:30:02.172492 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:02.172466    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs" event={"ID":"dd106df6-76b0-4919-8ec1-3f34d9da3d02","Type":"ContainerDied","Data":"aaeb33152409eddbce6e747d80edd34e4f41974e0fbc5640a33a27258e9a8f66"}
Oct 30 04:30:02.172647 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:02.172631    5072 scope.go:122] "RemoveContainer" containerID="aaeb33152409eddbce6e747d80edd34e4f41974e0fbc5640a33a27258e9a8f66"
Oct 30 04:30:02.174263 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:02.174242    5072 generic.go:356] "Generic (PLEG): container finished" podID="cafe1f28-6810-4682-84fa-a31518286947" containerID="f76e67d4ee8c90fa8ac83926c7ca71fafb8c114464dcc4c33f9e77168d6cffec" exitCode=60
Oct 30 04:30:02.174337 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:02.174281    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz" event={"ID":"cafe1f28-6810-4682-84fa-a31518286947","Type":"ContainerDied","Data":"f76e67d4ee8c90fa8ac83926c7ca71fafb8c114464dcc4c33f9e77168d6cffec"}
Oct 30 04:30:02.174457 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:02.174443    5072 scope.go:122] "RemoveContainer" containerID="f76e67d4ee8c90fa8ac83926c7ca71fafb8c114464dcc4c33f9e77168d6cffec"
Oct 30 04:30:03.184103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:03.183451    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x" event={"ID":"5df1e95a-34b9-45ac-beb6-523c9de84f05","Type":"ContainerStarted","Data":"17fa5d7303b6f90b98862e99022fad52d9a4edc30f7780549556436a13ed1385"}
Oct 30 04:30:03.185682 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:03.185656    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs" event={"ID":"dd106df6-76b0-4919-8ec1-3f34d9da3d02","Type":"ContainerStarted","Data":"3481b3860ad2efb723f72635064829c233ce61c683a847ac247c0dea83ca9658"}
Oct 30 04:30:03.187574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:03.187550    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz" event={"ID":"cafe1f28-6810-4682-84fa-a31518286947","Type":"ContainerStarted","Data":"7680cc79ebe43b2dc6e57ecd97f44667e1a241886e706906fe3318a1e548aba4"}
Oct 30 04:30:03.189285 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:03.189264    5072 generic.go:356] "Generic (PLEG): container finished" podID="657b39ad-f9e4-4635-a144-3f434f7a193e" containerID="13c45c6859ae657af96977414733338a1971d6ee700107e07cd9dd56c907a219" exitCode=60
Oct 30 04:30:03.189357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:03.189300    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465" event={"ID":"657b39ad-f9e4-4635-a144-3f434f7a193e","Type":"ContainerDied","Data":"13c45c6859ae657af96977414733338a1971d6ee700107e07cd9dd56c907a219"}
Oct 30 04:30:03.189357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:03.189332    5072 scope.go:122] "RemoveContainer" containerID="33a3db1481991849ff47c637ad3c7943e226b15fde8d09cadc908f9cc3420e0b"
Oct 30 04:30:03.189515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:03.189499    5072 scope.go:122] "RemoveContainer" containerID="13c45c6859ae657af96977414733338a1971d6ee700107e07cd9dd56c907a219"
Oct 30 04:30:03.189752 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:03.189734    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465_odh-ai-gateway-infra(657b39ad-f9e4-4635-a144-3f434f7a193e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e"
Oct 30 04:30:03.267945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:03.267900    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz" podStartSLOduration=3.26788729 podStartE2EDuration="3.26788729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:03.267477408 +0000 UTC m=+1983.008900506" watchObservedRunningTime="2026-10-30 04:30:03.26788729 +0000 UTC m=+1983.009310386"
Oct 30 04:30:03.270412 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:03.270376    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x" podStartSLOduration=3.270366704 podStartE2EDuration="3.270366704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:03.225717641 +0000 UTC m=+1982.967140737" watchObservedRunningTime="2026-10-30 04:30:03.270366704 +0000 UTC m=+1983.011789794"
Oct 30 04:30:03.312414 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:03.312252    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs" podStartSLOduration=3.312232298 podStartE2EDuration="3.312232298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:03.308984952 +0000 UTC m=+1983.050408049" watchObservedRunningTime="2026-10-30 04:30:03.312232298 +0000 UTC m=+1983.053655396"
Oct 30 04:30:04.197324 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.197298    5072 scope.go:122] "RemoveContainer" containerID="13c45c6859ae657af96977414733338a1971d6ee700107e07cd9dd56c907a219"
Oct 30 04:30:04.197777 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:04.197581    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465_odh-ai-gateway-infra(657b39ad-f9e4-4635-a144-3f434f7a193e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e"
Oct 30 04:30:04.199042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.199018    5072 generic.go:356] "Generic (PLEG): container finished" podID="5df1e95a-34b9-45ac-beb6-523c9de84f05" containerID="17fa5d7303b6f90b98862e99022fad52d9a4edc30f7780549556436a13ed1385" exitCode=60
Oct 30 04:30:04.199175 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.199122    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x" event={"ID":"5df1e95a-34b9-45ac-beb6-523c9de84f05","Type":"ContainerDied","Data":"17fa5d7303b6f90b98862e99022fad52d9a4edc30f7780549556436a13ed1385"}
Oct 30 04:30:04.199261 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.199187    5072 scope.go:122] "RemoveContainer" containerID="8272c95f774c72de8927bd6f6679201420f9dcbe91d5693c4bcae8fd20a12a84"
Oct 30 04:30:04.199350 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.199333    5072 scope.go:122] "RemoveContainer" containerID="17fa5d7303b6f90b98862e99022fad52d9a4edc30f7780549556436a13ed1385"
Oct 30 04:30:04.199631 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:04.199596    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888910-qwp8x_odh-ai-gateway-infra(5df1e95a-34b9-45ac-beb6-523c9de84f05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05"
Oct 30 04:30:04.201059 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.201039    5072 generic.go:356] "Generic (PLEG): container finished" podID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" containerID="3481b3860ad2efb723f72635064829c233ce61c683a847ac247c0dea83ca9658" exitCode=60
Oct 30 04:30:04.201154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.201108    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs" event={"ID":"dd106df6-76b0-4919-8ec1-3f34d9da3d02","Type":"ContainerDied","Data":"3481b3860ad2efb723f72635064829c233ce61c683a847ac247c0dea83ca9658"}
Oct 30 04:30:04.201340 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.201327    5072 scope.go:122] "RemoveContainer" containerID="3481b3860ad2efb723f72635064829c233ce61c683a847ac247c0dea83ca9658"
Oct 30 04:30:04.201573 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:04.201551    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs_odh-ai-gateway-infra(dd106df6-76b0-4919-8ec1-3f34d9da3d02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02"
Oct 30 04:30:04.202867 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.202848    5072 generic.go:356] "Generic (PLEG): container finished" podID="cafe1f28-6810-4682-84fa-a31518286947" containerID="7680cc79ebe43b2dc6e57ecd97f44667e1a241886e706906fe3318a1e548aba4" exitCode=60
Oct 30 04:30:04.202948 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.202899    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz" event={"ID":"cafe1f28-6810-4682-84fa-a31518286947","Type":"ContainerDied","Data":"7680cc79ebe43b2dc6e57ecd97f44667e1a241886e706906fe3318a1e548aba4"}
Oct 30 04:30:04.203110 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.203097    5072 scope.go:122] "RemoveContainer" containerID="7680cc79ebe43b2dc6e57ecd97f44667e1a241886e706906fe3318a1e548aba4"
Oct 30 04:30:04.203324 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:04.203307    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz_odh-ai-gateway-infra(cafe1f28-6810-4682-84fa-a31518286947)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz" podUID="cafe1f28-6810-4682-84fa-a31518286947"
Oct 30 04:30:04.212692 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.212674    5072 scope.go:122] "RemoveContainer" containerID="aaeb33152409eddbce6e747d80edd34e4f41974e0fbc5640a33a27258e9a8f66"
Oct 30 04:30:04.225097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:04.225068    5072 scope.go:122] "RemoveContainer" containerID="f76e67d4ee8c90fa8ac83926c7ca71fafb8c114464dcc4c33f9e77168d6cffec"
Oct 30 04:30:05.211835 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:05.211809    5072 scope.go:122] "RemoveContainer" containerID="7680cc79ebe43b2dc6e57ecd97f44667e1a241886e706906fe3318a1e548aba4"
Oct 30 04:30:05.212304 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:05.212094    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz_odh-ai-gateway-infra(cafe1f28-6810-4682-84fa-a31518286947)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz" podUID="cafe1f28-6810-4682-84fa-a31518286947"
Oct 30 04:30:05.213612 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:05.213594    5072 scope.go:122] "RemoveContainer" containerID="17fa5d7303b6f90b98862e99022fad52d9a4edc30f7780549556436a13ed1385"
Oct 30 04:30:05.213859 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:05.213840    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888910-qwp8x_odh-ai-gateway-infra(5df1e95a-34b9-45ac-beb6-523c9de84f05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05"
Oct 30 04:30:05.215043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:05.215025    5072 scope.go:122] "RemoveContainer" containerID="3481b3860ad2efb723f72635064829c233ce61c683a847ac247c0dea83ca9658"
Oct 30 04:30:05.215264 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:05.215248    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs_odh-ai-gateway-infra(dd106df6-76b0-4919-8ec1-3f34d9da3d02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02"
Oct 30 04:30:11.754793 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:11.754748    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:30:15.400819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:15.400792    5072 scope.go:122] "RemoveContainer" containerID="3481b3860ad2efb723f72635064829c233ce61c683a847ac247c0dea83ca9658"
Oct 30 04:30:16.287508 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:16.287479    5072 generic.go:356] "Generic (PLEG): container finished" podID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" containerID="c922345a96b604c591db1fc92817b62f24a17b3ff0bf9da0f5f70c86fa63c8e4" exitCode=60
Oct 30 04:30:16.287652 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:16.287559    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs" event={"ID":"dd106df6-76b0-4919-8ec1-3f34d9da3d02","Type":"ContainerDied","Data":"c922345a96b604c591db1fc92817b62f24a17b3ff0bf9da0f5f70c86fa63c8e4"}
Oct 30 04:30:16.287652 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:16.287621    5072 scope.go:122] "RemoveContainer" containerID="3481b3860ad2efb723f72635064829c233ce61c683a847ac247c0dea83ca9658"
Oct 30 04:30:16.288018 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:16.287995    5072 scope.go:122] "RemoveContainer" containerID="c922345a96b604c591db1fc92817b62f24a17b3ff0bf9da0f5f70c86fa63c8e4"
Oct 30 04:30:16.288307 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:16.288289    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs_odh-ai-gateway-infra(dd106df6-76b0-4919-8ec1-3f34d9da3d02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02"
Oct 30 04:30:17.352999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:17.352967    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"]
Oct 30 04:30:17.400742 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:17.400713    5072 scope.go:122] "RemoveContainer" containerID="13c45c6859ae657af96977414733338a1971d6ee700107e07cd9dd56c907a219"
Oct 30 04:30:17.400928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:17.400845    5072 scope.go:122] "RemoveContainer" containerID="17fa5d7303b6f90b98862e99022fad52d9a4edc30f7780549556436a13ed1385"
Oct 30 04:30:17.401384 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:17.401362    5072 scope.go:122] "RemoveContainer" containerID="7680cc79ebe43b2dc6e57ecd97f44667e1a241886e706906fe3318a1e548aba4"
Oct 30 04:30:17.526160 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:17.526138    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"
Oct 30 04:30:17.711546 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:17.711521    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd106df6-76b0-4919-8ec1-3f34d9da3d02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd106df6-76b0-4919-8ec1-3f34d9da3d02-maas-api-service-ca\") pod \"dd106df6-76b0-4919-8ec1-3f34d9da3d02\" (UID: \"dd106df6-76b0-4919-8ec1-3f34d9da3d02\") "
Oct 30 04:30:17.711672 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:17.711649    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd106df6-76b0-4919-8ec1-3f34d9da3d02-kube-api-access-zk9wz\" (UniqueName: \"kubernetes.io/projected/dd106df6-76b0-4919-8ec1-3f34d9da3d02-kube-api-access-zk9wz\") pod \"dd106df6-76b0-4919-8ec1-3f34d9da3d02\" (UID: \"dd106df6-76b0-4919-8ec1-3f34d9da3d02\") "
Oct 30 04:30:17.711856 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:17.711832    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd106df6-76b0-4919-8ec1-3f34d9da3d02-maas-api-service-ca" pod "dd106df6-76b0-4919-8ec1-3f34d9da3d02" (UID: "dd106df6-76b0-4919-8ec1-3f34d9da3d02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:30:17.711953 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:17.711920    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd106df6-76b0-4919-8ec1-3f34d9da3d02-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:30:17.713809 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:17.713791    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd106df6-76b0-4919-8ec1-3f34d9da3d02-kube-api-access-zk9wz" pod "dd106df6-76b0-4919-8ec1-3f34d9da3d02" (UID: "dd106df6-76b0-4919-8ec1-3f34d9da3d02"). InnerVolumeSpecName "kube-api-access-zk9wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:30:17.812982 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:17.812955    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk9wz\" (UniqueName: \"kubernetes.io/projected/dd106df6-76b0-4919-8ec1-3f34d9da3d02-kube-api-access-zk9wz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:30:18.306116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:18.306059    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz" event={"ID":"cafe1f28-6810-4682-84fa-a31518286947","Type":"ContainerStarted","Data":"95158d91d6540732c675bcdf9265a1bc0a416f7528d5d34726970a58db126afe"}
Oct 30 04:30:18.308416 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:18.308386    5072 generic.go:356] "Generic (PLEG): container finished" podID="657b39ad-f9e4-4635-a144-3f434f7a193e" containerID="bed864f5a654f0668586f7559cd49e624285b1de89f35c20fc725ac2ea77dd13" exitCode=60
Oct 30 04:30:18.308553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:18.308460    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465" event={"ID":"657b39ad-f9e4-4635-a144-3f434f7a193e","Type":"ContainerDied","Data":"bed864f5a654f0668586f7559cd49e624285b1de89f35c20fc725ac2ea77dd13"}
Oct 30 04:30:18.308553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:18.308518    5072 scope.go:122] "RemoveContainer" containerID="13c45c6859ae657af96977414733338a1971d6ee700107e07cd9dd56c907a219"
Oct 30 04:30:18.308905 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:18.308853    5072 scope.go:122] "RemoveContainer" containerID="bed864f5a654f0668586f7559cd49e624285b1de89f35c20fc725ac2ea77dd13"
Oct 30 04:30:18.309244 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:18.309210    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465_odh-ai-gateway-infra(657b39ad-f9e4-4635-a144-3f434f7a193e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e"
Oct 30 04:30:18.310915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:18.310893    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x" event={"ID":"5df1e95a-34b9-45ac-beb6-523c9de84f05","Type":"ContainerStarted","Data":"0bef278023204a5b6f9cb9897d4ee835bc61f7713b128c1f5f6e86256193913a"}
Oct 30 04:30:18.312437 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:18.312413    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs" event={"ID":"dd106df6-76b0-4919-8ec1-3f34d9da3d02","Type":"ContainerDied","Data":"bd531e24d6f90c087f2a3e3e0fac61219b5b76b42707dfad423d339efec29b81"}
Oct 30 04:30:18.312540 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:18.312425    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"
Oct 30 04:30:18.321978 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:18.321961    5072 scope.go:122] "RemoveContainer" containerID="c922345a96b604c591db1fc92817b62f24a17b3ff0bf9da0f5f70c86fa63c8e4"
Oct 30 04:30:18.391750 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:18.391724    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"]
Oct 30 04:30:18.407757 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:18.407725    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888910-hhkqs"]
Oct 30 04:30:19.319917 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.319883    5072 generic.go:356] "Generic (PLEG): container finished" podID="5df1e95a-34b9-45ac-beb6-523c9de84f05" containerID="0bef278023204a5b6f9cb9897d4ee835bc61f7713b128c1f5f6e86256193913a" exitCode=60
Oct 30 04:30:19.320103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.319956    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x" event={"ID":"5df1e95a-34b9-45ac-beb6-523c9de84f05","Type":"ContainerDied","Data":"0bef278023204a5b6f9cb9897d4ee835bc61f7713b128c1f5f6e86256193913a"}
Oct 30 04:30:19.320103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.320024    5072 scope.go:122] "RemoveContainer" containerID="17fa5d7303b6f90b98862e99022fad52d9a4edc30f7780549556436a13ed1385"
Oct 30 04:30:19.320502 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.320461    5072 scope.go:122] "RemoveContainer" containerID="0bef278023204a5b6f9cb9897d4ee835bc61f7713b128c1f5f6e86256193913a"
Oct 30 04:30:19.320786 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:19.320768    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888910-qwp8x_odh-ai-gateway-infra(5df1e95a-34b9-45ac-beb6-523c9de84f05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05"
Oct 30 04:30:19.323031 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.323011    5072 generic.go:356] "Generic (PLEG): container finished" podID="cafe1f28-6810-4682-84fa-a31518286947" containerID="95158d91d6540732c675bcdf9265a1bc0a416f7528d5d34726970a58db126afe" exitCode=60
Oct 30 04:30:19.323135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.323064    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz" event={"ID":"cafe1f28-6810-4682-84fa-a31518286947","Type":"ContainerDied","Data":"95158d91d6540732c675bcdf9265a1bc0a416f7528d5d34726970a58db126afe"}
Oct 30 04:30:19.323459 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.323446    5072 scope.go:122] "RemoveContainer" containerID="95158d91d6540732c675bcdf9265a1bc0a416f7528d5d34726970a58db126afe"
Oct 30 04:30:19.324456 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:19.324424    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz_odh-ai-gateway-infra(cafe1f28-6810-4682-84fa-a31518286947)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz" podUID="cafe1f28-6810-4682-84fa-a31518286947"
Oct 30 04:30:19.344858 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.344840    5072 scope.go:122] "RemoveContainer" containerID="7680cc79ebe43b2dc6e57ecd97f44667e1a241886e706906fe3318a1e548aba4"
Oct 30 04:30:19.367779 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.367754    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"]
Oct 30 04:30:19.451093 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.451051    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"]
Oct 30 04:30:19.497415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.497385    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"]
Oct 30 04:30:19.615951 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.615930    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"
Oct 30 04:30:19.639841 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.639815    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657b39ad-f9e4-4635-a144-3f434f7a193e-kube-api-access-k4sc7\" (UniqueName: \"kubernetes.io/projected/657b39ad-f9e4-4635-a144-3f434f7a193e-kube-api-access-k4sc7\") pod \"657b39ad-f9e4-4635-a144-3f434f7a193e\" (UID: \"657b39ad-f9e4-4635-a144-3f434f7a193e\") "
Oct 30 04:30:19.639943 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.639907    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657b39ad-f9e4-4635-a144-3f434f7a193e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657b39ad-f9e4-4635-a144-3f434f7a193e-maas-api-service-ca\") pod \"657b39ad-f9e4-4635-a144-3f434f7a193e\" (UID: \"657b39ad-f9e4-4635-a144-3f434f7a193e\") "
Oct 30 04:30:19.640230 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.640211    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657b39ad-f9e4-4635-a144-3f434f7a193e-maas-api-service-ca" pod "657b39ad-f9e4-4635-a144-3f434f7a193e" (UID: "657b39ad-f9e4-4635-a144-3f434f7a193e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:30:19.640306 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.640291    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657b39ad-f9e4-4635-a144-3f434f7a193e-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:30:19.641922 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.641902    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657b39ad-f9e4-4635-a144-3f434f7a193e-kube-api-access-k4sc7" pod "657b39ad-f9e4-4635-a144-3f434f7a193e" (UID: "657b39ad-f9e4-4635-a144-3f434f7a193e"). InnerVolumeSpecName "kube-api-access-k4sc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:30:19.741415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:19.741383    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4sc7\" (UniqueName: \"kubernetes.io/projected/657b39ad-f9e4-4635-a144-3f434f7a193e-kube-api-access-k4sc7\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:30:20.343517 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.343491    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"
Oct 30 04:30:20.343697 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.343526    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465" event={"ID":"657b39ad-f9e4-4635-a144-3f434f7a193e","Type":"ContainerDied","Data":"d9feaf5e97c8cd4b6d9414ef6766325c70884de72418d88638b92ad36946a3c6"}
Oct 30 04:30:20.343697 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.343598    5072 scope.go:122] "RemoveContainer" containerID="bed864f5a654f0668586f7559cd49e624285b1de89f35c20fc725ac2ea77dd13"
Oct 30 04:30:20.411457 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.411425    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" path="/var/lib/kubelet/pods/dd106df6-76b0-4919-8ec1-3f34d9da3d02/volumes"
Oct 30 04:30:20.499130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.499098    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"]
Oct 30 04:30:20.507711 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.507688    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888910-wl465"]
Oct 30 04:30:20.541744 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.541718    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"
Oct 30 04:30:20.545437 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.545415    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"
Oct 30 04:30:20.552844 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.552823    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafe1f28-6810-4682-84fa-a31518286947-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafe1f28-6810-4682-84fa-a31518286947-maas-api-service-ca\") pod \"cafe1f28-6810-4682-84fa-a31518286947\" (UID: \"cafe1f28-6810-4682-84fa-a31518286947\") "
Oct 30 04:30:20.552915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.552858    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df1e95a-34b9-45ac-beb6-523c9de84f05-kube-api-access-j2vx8\" (UniqueName: \"kubernetes.io/projected/5df1e95a-34b9-45ac-beb6-523c9de84f05-kube-api-access-j2vx8\") pod \"5df1e95a-34b9-45ac-beb6-523c9de84f05\" (UID: \"5df1e95a-34b9-45ac-beb6-523c9de84f05\") "
Oct 30 04:30:20.552965 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.552935    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafe1f28-6810-4682-84fa-a31518286947-kube-api-access-5snw4\" (UniqueName: \"kubernetes.io/projected/cafe1f28-6810-4682-84fa-a31518286947-kube-api-access-5snw4\") pod \"cafe1f28-6810-4682-84fa-a31518286947\" (UID: \"cafe1f28-6810-4682-84fa-a31518286947\") "
Oct 30 04:30:20.552997 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.552960    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df1e95a-34b9-45ac-beb6-523c9de84f05-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df1e95a-34b9-45ac-beb6-523c9de84f05-maas-api-service-ca\") pod \"5df1e95a-34b9-45ac-beb6-523c9de84f05\" (UID: \"5df1e95a-34b9-45ac-beb6-523c9de84f05\") "
Oct 30 04:30:20.553247 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.553220    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafe1f28-6810-4682-84fa-a31518286947-maas-api-service-ca" pod "cafe1f28-6810-4682-84fa-a31518286947" (UID: "cafe1f28-6810-4682-84fa-a31518286947"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:30:20.553373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.553352    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafe1f28-6810-4682-84fa-a31518286947-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:30:20.553611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.553558    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df1e95a-34b9-45ac-beb6-523c9de84f05-maas-api-service-ca" pod "5df1e95a-34b9-45ac-beb6-523c9de84f05" (UID: "5df1e95a-34b9-45ac-beb6-523c9de84f05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:30:20.555253 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.555223    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafe1f28-6810-4682-84fa-a31518286947-kube-api-access-5snw4" pod "cafe1f28-6810-4682-84fa-a31518286947" (UID: "cafe1f28-6810-4682-84fa-a31518286947"). InnerVolumeSpecName "kube-api-access-5snw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:30:20.555342 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.555302    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df1e95a-34b9-45ac-beb6-523c9de84f05-kube-api-access-j2vx8" pod "5df1e95a-34b9-45ac-beb6-523c9de84f05" (UID: "5df1e95a-34b9-45ac-beb6-523c9de84f05"). InnerVolumeSpecName "kube-api-access-j2vx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:30:20.654509 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.654439    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2vx8\" (UniqueName: \"kubernetes.io/projected/5df1e95a-34b9-45ac-beb6-523c9de84f05-kube-api-access-j2vx8\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:30:20.654509 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.654474    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5snw4\" (UniqueName: \"kubernetes.io/projected/cafe1f28-6810-4682-84fa-a31518286947-kube-api-access-5snw4\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:30:20.654509 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:20.654488    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df1e95a-34b9-45ac-beb6-523c9de84f05-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:30:21.360415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:21.360359    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x" event={"ID":"5df1e95a-34b9-45ac-beb6-523c9de84f05","Type":"ContainerDied","Data":"037d5ce6138368dbbff7e9f96e38fa2fc1261af3acf4721f8e2b00d6a7a81f05"}
Oct 30 04:30:21.360415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:21.360422    5072 scope.go:122] "RemoveContainer" containerID="0bef278023204a5b6f9cb9897d4ee835bc61f7713b128c1f5f6e86256193913a"
Oct 30 04:30:21.360792 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:21.360538    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"
Oct 30 04:30:21.364969 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:21.364945    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz" event={"ID":"cafe1f28-6810-4682-84fa-a31518286947","Type":"ContainerDied","Data":"15c38255416d09beb0ec18d9e4c3ddc94fab2f29c4dbb567954837ad1cc35519"}
Oct 30 04:30:21.365106 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:21.365034    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"
Oct 30 04:30:21.375754 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:21.375726    5072 scope.go:122] "RemoveContainer" containerID="95158d91d6540732c675bcdf9265a1bc0a416f7528d5d34726970a58db126afe"
Oct 30 04:30:21.417936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:21.417907    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"]
Oct 30 04:30:21.423835 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:21.423810    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888910-qwp8x"]
Oct 30 04:30:21.467018 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:21.466964    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"]
Oct 30 04:30:21.469571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:21.469543    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888910-tlqlz"]
Oct 30 04:30:22.409240 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:22.409205    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05" path="/var/lib/kubelet/pods/5df1e95a-34b9-45ac-beb6-523c9de84f05/volumes"
Oct 30 04:30:22.409989 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:22.409969    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e" path="/var/lib/kubelet/pods/657b39ad-f9e4-4635-a144-3f434f7a193e/volumes"
Oct 30 04:30:22.410526 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:22.410508    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafe1f28-6810-4682-84fa-a31518286947" path="/var/lib/kubelet/pods/cafe1f28-6810-4682-84fa-a31518286947/volumes"
Oct 30 04:30:29.819658 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.819618    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"]
Oct 30 04:30:29.821655 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.821624    5072 cpu_manager.go:414] "containerMap: removing container" podUID="cafe1f28-6810-4682-84fa-a31518286947" containerName="cleanup"
Oct 30 04:30:29.821850 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.821823    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafe1f28-6810-4682-84fa-a31518286947" containerName="cleanup"
Oct 30 04:30:29.821997 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.821885    5072 cpu_manager.go:414] "containerMap: removing container" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" containerName="cleanup"
Oct 30 04:30:29.821997 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.821894    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" containerName="cleanup"
Oct 30 04:30:29.821997 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.821961    5072 cpu_manager.go:414] "containerMap: removing container" podUID="cafe1f28-6810-4682-84fa-a31518286947" containerName="cleanup"
Oct 30 04:30:29.821997 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.821969    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafe1f28-6810-4682-84fa-a31518286947" containerName="cleanup"
Oct 30 04:30:29.821997 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.821988    5072 cpu_manager.go:414] "containerMap: removing container" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e" containerName="cleanup"
Oct 30 04:30:29.821997 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.821996    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e" containerName="cleanup"
Oct 30 04:30:29.822315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822018    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05" containerName="cleanup"
Oct 30 04:30:29.822315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822037    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05" containerName="cleanup"
Oct 30 04:30:29.822315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822069    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05" containerName="cleanup"
Oct 30 04:30:29.822315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822104    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05" containerName="cleanup"
Oct 30 04:30:29.822315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822129    5072 cpu_manager.go:414] "containerMap: removing container" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" containerName="cleanup"
Oct 30 04:30:29.822315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822136    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" containerName="cleanup"
Oct 30 04:30:29.822315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822164    5072 cpu_manager.go:414] "containerMap: removing container" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e" containerName="cleanup"
Oct 30 04:30:29.822315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822172    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e" containerName="cleanup"
Oct 30 04:30:29.822695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822441    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05" containerName="cleanup"
Oct 30 04:30:29.822695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822455    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" containerName="cleanup"
Oct 30 04:30:29.822695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822463    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e" containerName="cleanup"
Oct 30 04:30:29.822695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822477    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" containerName="cleanup"
Oct 30 04:30:29.822695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822490    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafe1f28-6810-4682-84fa-a31518286947" containerName="cleanup"
Oct 30 04:30:29.822695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822506    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafe1f28-6810-4682-84fa-a31518286947" containerName="cleanup"
Oct 30 04:30:29.822695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822518    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" containerName="cleanup"
Oct 30 04:30:29.822695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822545    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e" containerName="cleanup"
Oct 30 04:30:29.822695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.822554    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05" containerName="cleanup"
Oct 30 04:30:29.869578 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.869339    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"]
Oct 30 04:30:29.869936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.869914    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"
Oct 30 04:30:29.885726 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.885706    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6a777c-dockercfg-9jz9c\""
Oct 30 04:30:29.885855 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.885731    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6a777c\""
Oct 30 04:30:29.999468 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.999430    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc5k6\" (UniqueName: \"kubernetes.io/projected/94c893d7-e177-4f52-b332-b17ce1cc8d9d-kube-api-access-hc5k6\") pod \"payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2\" (UID: \"94c893d7-e177-4f52-b332-b17ce1cc8d9d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"
Oct 30 04:30:29.999678 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:29.999645    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c893d7-e177-4f52-b332-b17ce1cc8d9d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2\" (UID: \"94c893d7-e177-4f52-b332-b17ce1cc8d9d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"
Oct 30 04:30:30.100798 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:30.100707    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc5k6\" (UniqueName: \"kubernetes.io/projected/94c893d7-e177-4f52-b332-b17ce1cc8d9d-kube-api-access-hc5k6\") pod \"payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2\" (UID: \"94c893d7-e177-4f52-b332-b17ce1cc8d9d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"
Oct 30 04:30:30.100974 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:30.100853    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c893d7-e177-4f52-b332-b17ce1cc8d9d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2\" (UID: \"94c893d7-e177-4f52-b332-b17ce1cc8d9d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"
Oct 30 04:30:30.101679 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:30.101657    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c893d7-e177-4f52-b332-b17ce1cc8d9d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2\" (UID: \"94c893d7-e177-4f52-b332-b17ce1cc8d9d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"
Oct 30 04:30:30.124933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:30.124902    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc5k6\" (UniqueName: \"kubernetes.io/projected/94c893d7-e177-4f52-b332-b17ce1cc8d9d-kube-api-access-hc5k6\") pod \"payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2\" (UID: \"94c893d7-e177-4f52-b332-b17ce1cc8d9d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"
Oct 30 04:30:30.188649 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:30.188620    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"
Oct 30 04:30:30.496167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:30.496136    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"]
Oct 30 04:30:30.497173 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:30:30.497143    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c893d7_e177_4f52_b332_b17ce1cc8d9d.slice/crio-bd2fb2f5fe443ac362935afc057c4b0a22bc3224f8ae6eec27e6e86cb9d58e7d WatchSource:0}: Error finding container bd2fb2f5fe443ac362935afc057c4b0a22bc3224f8ae6eec27e6e86cb9d58e7d: Status 404 returned error can't find the container with id bd2fb2f5fe443ac362935afc057c4b0a22bc3224f8ae6eec27e6e86cb9d58e7d
Oct 30 04:30:31.431300 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:31.431268    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2" event={"ID":"94c893d7-e177-4f52-b332-b17ce1cc8d9d","Type":"ContainerStarted","Data":"bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3"}
Oct 30 04:30:31.431703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:31.431319    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2" event={"ID":"94c893d7-e177-4f52-b332-b17ce1cc8d9d","Type":"ContainerStarted","Data":"bd2fb2f5fe443ac362935afc057c4b0a22bc3224f8ae6eec27e6e86cb9d58e7d"}
Oct 30 04:30:31.431703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:31.431399    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"
Oct 30 04:30:31.486435 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:31.485928    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2" podStartSLOduration=2.48590961 podStartE2EDuration="2.48590961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:31.481557422 +0000 UTC m=+2011.222980520" watchObservedRunningTime="2026-10-30 04:30:31.48590961 +0000 UTC m=+2011.227332708"
Oct 30 04:30:41.752242 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:30:41.752158    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:30:42.440304 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:42.440276    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"
Oct 30 04:30:48.038225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.038193    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"]
Oct 30 04:30:48.038725 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.038712    5072 cpu_manager.go:414] "containerMap: removing container" podUID="cafe1f28-6810-4682-84fa-a31518286947" containerName="cleanup"
Oct 30 04:30:48.038769 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.038726    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafe1f28-6810-4682-84fa-a31518286947" containerName="cleanup"
Oct 30 04:30:48.038769 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.038744    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05" containerName="cleanup"
Oct 30 04:30:48.038769 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.038749    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05" containerName="cleanup"
Oct 30 04:30:48.038769 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.038767    5072 cpu_manager.go:414] "containerMap: removing container" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e" containerName="cleanup"
Oct 30 04:30:48.038894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.038773    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e" containerName="cleanup"
Oct 30 04:30:48.038894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.038787    5072 cpu_manager.go:414] "containerMap: removing container" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" containerName="cleanup"
Oct 30 04:30:48.038894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.038791    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd106df6-76b0-4919-8ec1-3f34d9da3d02" containerName="cleanup"
Oct 30 04:30:48.038894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.038878    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df1e95a-34b9-45ac-beb6-523c9de84f05" containerName="cleanup"
Oct 30 04:30:48.038894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.038889    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafe1f28-6810-4682-84fa-a31518286947" containerName="cleanup"
Oct 30 04:30:48.039036 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.038900    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="657b39ad-f9e4-4635-a144-3f434f7a193e" containerName="cleanup"
Oct 30 04:30:48.043522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.043494    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"
Oct 30 04:30:48.061045 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.061021    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"]
Oct 30 04:30:48.070942 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.070916    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-eb9d23-dockercfg-vmjqg\""
Oct 30 04:30:48.071062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.070956    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-eb9d23\""
Oct 30 04:30:48.087606 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.087573    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"]
Oct 30 04:30:48.094440 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.094417    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"
Oct 30 04:30:48.136296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.134641    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b234059b-84a5-423f-b33e-592ca68d9fe0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh\" (UID: \"b234059b-84a5-423f-b33e-592ca68d9fe0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"
Oct 30 04:30:48.136296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.134731    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gdrt\" (UniqueName: \"kubernetes.io/projected/1645d371-14e9-4c69-be15-cfd189696baf-kube-api-access-7gdrt\") pod \"payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx\" (UID: \"1645d371-14e9-4c69-be15-cfd189696baf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"
Oct 30 04:30:48.136296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.134832    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1645d371-14e9-4c69-be15-cfd189696baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx\" (UID: \"1645d371-14e9-4c69-be15-cfd189696baf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"
Oct 30 04:30:48.136296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.134875    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5b7v\" (UniqueName: \"kubernetes.io/projected/b234059b-84a5-423f-b33e-592ca68d9fe0-kube-api-access-l5b7v\") pod \"payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh\" (UID: \"b234059b-84a5-423f-b33e-592ca68d9fe0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"
Oct 30 04:30:48.145275 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.145246    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"]
Oct 30 04:30:48.235974 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.235924    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b234059b-84a5-423f-b33e-592ca68d9fe0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh\" (UID: \"b234059b-84a5-423f-b33e-592ca68d9fe0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"
Oct 30 04:30:48.235974 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.235970    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gdrt\" (UniqueName: \"kubernetes.io/projected/1645d371-14e9-4c69-be15-cfd189696baf-kube-api-access-7gdrt\") pod \"payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx\" (UID: \"1645d371-14e9-4c69-be15-cfd189696baf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"
Oct 30 04:30:48.236259 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.236022    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1645d371-14e9-4c69-be15-cfd189696baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx\" (UID: \"1645d371-14e9-4c69-be15-cfd189696baf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"
Oct 30 04:30:48.236259 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.236045    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5b7v\" (UniqueName: \"kubernetes.io/projected/b234059b-84a5-423f-b33e-592ca68d9fe0-kube-api-access-l5b7v\") pod \"payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh\" (UID: \"b234059b-84a5-423f-b33e-592ca68d9fe0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"
Oct 30 04:30:48.236805 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.236778    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b234059b-84a5-423f-b33e-592ca68d9fe0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh\" (UID: \"b234059b-84a5-423f-b33e-592ca68d9fe0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"
Oct 30 04:30:48.236936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.236787    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1645d371-14e9-4c69-be15-cfd189696baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx\" (UID: \"1645d371-14e9-4c69-be15-cfd189696baf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"
Oct 30 04:30:48.260288 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.260259    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gdrt\" (UniqueName: \"kubernetes.io/projected/1645d371-14e9-4c69-be15-cfd189696baf-kube-api-access-7gdrt\") pod \"payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx\" (UID: \"1645d371-14e9-4c69-be15-cfd189696baf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"
Oct 30 04:30:48.260381 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.260259    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5b7v\" (UniqueName: \"kubernetes.io/projected/b234059b-84a5-423f-b33e-592ca68d9fe0-kube-api-access-l5b7v\") pod \"payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh\" (UID: \"b234059b-84a5-423f-b33e-592ca68d9fe0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"
Oct 30 04:30:48.357587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.357537    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"
Oct 30 04:30:48.447418 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.447384    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"
Oct 30 04:30:48.661600 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.661559    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"]
Oct 30 04:30:48.662467 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:30:48.662444    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1645d371_14e9_4c69_be15_cfd189696baf.slice/crio-b6e15a433bac13c38ff6e357446f25b8c3cd8b98c914c168f12b53a3abf7824d WatchSource:0}: Error finding container b6e15a433bac13c38ff6e357446f25b8c3cd8b98c914c168f12b53a3abf7824d: Status 404 returned error can't find the container with id b6e15a433bac13c38ff6e357446f25b8c3cd8b98c914c168f12b53a3abf7824d
Oct 30 04:30:48.765888 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:48.765816    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"]
Oct 30 04:30:49.544151 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:49.544114    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx" event={"ID":"1645d371-14e9-4c69-be15-cfd189696baf","Type":"ContainerStarted","Data":"cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4"}
Oct 30 04:30:49.544668 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:49.544175    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx" event={"ID":"1645d371-14e9-4c69-be15-cfd189696baf","Type":"ContainerStarted","Data":"b6e15a433bac13c38ff6e357446f25b8c3cd8b98c914c168f12b53a3abf7824d"}
Oct 30 04:30:49.544668 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:49.544203    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"
Oct 30 04:30:49.546197 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:49.546166    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh" event={"ID":"b234059b-84a5-423f-b33e-592ca68d9fe0","Type":"ContainerStarted","Data":"578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2"}
Oct 30 04:30:49.546365 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:49.546216    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh" event={"ID":"b234059b-84a5-423f-b33e-592ca68d9fe0","Type":"ContainerStarted","Data":"ceed4ddfa31c10b752f2996634b1bc1f9a3b2b033cccbbda58449570da90cadf"}
Oct 30 04:30:49.546365 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:49.546260    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"
Oct 30 04:30:49.587767 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:49.587721    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx" podStartSLOduration=1.5877084350000001 podStartE2EDuration="1.587708435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:49.585385749 +0000 UTC m=+2029.326808847" watchObservedRunningTime="2026-10-30 04:30:49.587708435 +0000 UTC m=+2029.329131531"
Oct 30 04:30:49.628295 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:49.628250    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh" podStartSLOduration=1.628236714 podStartE2EDuration="1.628236714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:49.626406155 +0000 UTC m=+2029.367829255" watchObservedRunningTime="2026-10-30 04:30:49.628236714 +0000 UTC m=+2029.369659810"
Oct 30 04:30:53.632665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.632635    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"]
Oct 30 04:30:53.636330 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.636310    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"
Oct 30 04:30:53.656220 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.656199    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"]
Oct 30 04:30:53.664905 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.664881    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-05f827\""
Oct 30 04:30:53.665436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.665172    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-05f827-dockercfg-r45lt\""
Oct 30 04:30:53.700243 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.700204    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd801aac-9f06-4f1f-b5a4-093f42890475-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh\" (UID: \"dd801aac-9f06-4f1f-b5a4-093f42890475\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"
Oct 30 04:30:53.700243 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.700256    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ts94\" (UniqueName: \"kubernetes.io/projected/dd801aac-9f06-4f1f-b5a4-093f42890475-kube-api-access-6ts94\") pod \"payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh\" (UID: \"dd801aac-9f06-4f1f-b5a4-093f42890475\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"
Oct 30 04:30:53.741126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.736464    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"]
Oct 30 04:30:53.742641 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.742602    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"
Oct 30 04:30:53.770001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.769971    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"]
Oct 30 04:30:53.801562 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.801527    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzgvf\" (UniqueName: \"kubernetes.io/projected/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-kube-api-access-jzgvf\") pod \"payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5\" (UID: \"e7f883b1-9cb3-4f99-bc58-99541ce74e1e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"
Oct 30 04:30:53.801743 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.801591    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd801aac-9f06-4f1f-b5a4-093f42890475-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh\" (UID: \"dd801aac-9f06-4f1f-b5a4-093f42890475\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"
Oct 30 04:30:53.801743 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.801685    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ts94\" (UniqueName: \"kubernetes.io/projected/dd801aac-9f06-4f1f-b5a4-093f42890475-kube-api-access-6ts94\") pod \"payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh\" (UID: \"dd801aac-9f06-4f1f-b5a4-093f42890475\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"
Oct 30 04:30:53.801897 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.801856    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5\" (UID: \"e7f883b1-9cb3-4f99-bc58-99541ce74e1e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"
Oct 30 04:30:53.802543 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.802516    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd801aac-9f06-4f1f-b5a4-093f42890475-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh\" (UID: \"dd801aac-9f06-4f1f-b5a4-093f42890475\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"
Oct 30 04:30:53.826265 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.826230    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ts94\" (UniqueName: \"kubernetes.io/projected/dd801aac-9f06-4f1f-b5a4-093f42890475-kube-api-access-6ts94\") pod \"payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh\" (UID: \"dd801aac-9f06-4f1f-b5a4-093f42890475\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"
Oct 30 04:30:53.903170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.903103    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzgvf\" (UniqueName: \"kubernetes.io/projected/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-kube-api-access-jzgvf\") pod \"payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5\" (UID: \"e7f883b1-9cb3-4f99-bc58-99541ce74e1e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"
Oct 30 04:30:53.903306 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.903197    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5\" (UID: \"e7f883b1-9cb3-4f99-bc58-99541ce74e1e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"
Oct 30 04:30:53.903891 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.903872    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5\" (UID: \"e7f883b1-9cb3-4f99-bc58-99541ce74e1e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"
Oct 30 04:30:53.927385 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.927364    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzgvf\" (UniqueName: \"kubernetes.io/projected/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-kube-api-access-jzgvf\") pod \"payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5\" (UID: \"e7f883b1-9cb3-4f99-bc58-99541ce74e1e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"
Oct 30 04:30:53.950766 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:53.950232    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"
Oct 30 04:30:54.076216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:54.076177    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"
Oct 30 04:30:54.263800 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:30:54.263769    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd801aac_9f06_4f1f_b5a4_093f42890475.slice/crio-ac7d335d60376b18f22aa6bc2b6ea50f241e8c28a901dededecc6d2c0ba2dfd9 WatchSource:0}: Error finding container ac7d335d60376b18f22aa6bc2b6ea50f241e8c28a901dededecc6d2c0ba2dfd9: Status 404 returned error can't find the container with id ac7d335d60376b18f22aa6bc2b6ea50f241e8c28a901dededecc6d2c0ba2dfd9
Oct 30 04:30:54.264047 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:54.264021    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"]
Oct 30 04:30:54.382700 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:54.382670    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"]
Oct 30 04:30:54.383150 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:30:54.383127    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f883b1_9cb3_4f99_bc58_99541ce74e1e.slice/crio-c6c663e1253ec9daf4acf5ffecf71964da632c371eed126d4d3d966b09f25c9d WatchSource:0}: Error finding container c6c663e1253ec9daf4acf5ffecf71964da632c371eed126d4d3d966b09f25c9d: Status 404 returned error can't find the container with id c6c663e1253ec9daf4acf5ffecf71964da632c371eed126d4d3d966b09f25c9d
Oct 30 04:30:54.579789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:54.579688    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5" event={"ID":"e7f883b1-9cb3-4f99-bc58-99541ce74e1e","Type":"ContainerStarted","Data":"b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60"}
Oct 30 04:30:54.580519 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:54.580454    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5" event={"ID":"e7f883b1-9cb3-4f99-bc58-99541ce74e1e","Type":"ContainerStarted","Data":"c6c663e1253ec9daf4acf5ffecf71964da632c371eed126d4d3d966b09f25c9d"}
Oct 30 04:30:54.580519 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:54.580510    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"
Oct 30 04:30:54.582269 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:54.582243    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh" event={"ID":"dd801aac-9f06-4f1f-b5a4-093f42890475","Type":"ContainerStarted","Data":"882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad"}
Oct 30 04:30:54.582366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:54.582293    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh" event={"ID":"dd801aac-9f06-4f1f-b5a4-093f42890475","Type":"ContainerStarted","Data":"ac7d335d60376b18f22aa6bc2b6ea50f241e8c28a901dededecc6d2c0ba2dfd9"}
Oct 30 04:30:54.582945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:54.582926    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"
Oct 30 04:30:54.623526 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:54.623486    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5" podStartSLOduration=1.623473779 podStartE2EDuration="1.623473779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:54.622454768 +0000 UTC m=+2034.363877866" watchObservedRunningTime="2026-10-30 04:30:54.623473779 +0000 UTC m=+2034.364896876"
Oct 30 04:30:54.664956 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:30:54.664909    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh" podStartSLOduration=1.66489639 podStartE2EDuration="1.66489639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:54.662510097 +0000 UTC m=+2034.403933205" watchObservedRunningTime="2026-10-30 04:30:54.66489639 +0000 UTC m=+2034.406319487"
Oct 30 04:31:00.554168 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:00.554139    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"
Oct 30 04:31:00.554546 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:00.554208    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"
Oct 30 04:31:05.592293 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:05.592264    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"
Oct 30 04:31:06.597742 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:06.597710    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"
Oct 30 04:31:11.754482 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:31:11.754442    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:31:15.227553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.227519    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"]
Oct 30 04:31:15.233421 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.233394    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"
Oct 30 04:31:15.249328 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.249284    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"]
Oct 30 04:31:15.249328 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.249314    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-448a3a-dockercfg-qzfcv\""
Oct 30 04:31:15.258874 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.258848    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-448a3a\""
Oct 30 04:31:15.316452 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.316417    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"]
Oct 30 04:31:15.321686 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.321660    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"
Oct 30 04:31:15.342208 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.342177    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"]
Oct 30 04:31:15.369505 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.365730    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbts6\" (UniqueName: \"kubernetes.io/projected/4d43f997-ab21-4268-ade1-9407c9c2e22a-kube-api-access-hbts6\") pod \"payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w\" (UID: \"4d43f997-ab21-4268-ade1-9407c9c2e22a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"
Oct 30 04:31:15.369505 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.365852    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d43f997-ab21-4268-ade1-9407c9c2e22a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w\" (UID: \"4d43f997-ab21-4268-ade1-9407c9c2e22a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"
Oct 30 04:31:15.466745 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.466711    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d43f997-ab21-4268-ade1-9407c9c2e22a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w\" (UID: \"4d43f997-ab21-4268-ade1-9407c9c2e22a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"
Oct 30 04:31:15.466937 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.466814    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckfx2\" (UniqueName: \"kubernetes.io/projected/c75248e8-cb72-4866-bf44-e9fd5f0b0167-kube-api-access-ckfx2\") pod \"payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg\" (UID: \"c75248e8-cb72-4866-bf44-e9fd5f0b0167\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"
Oct 30 04:31:15.467000 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.466954    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbts6\" (UniqueName: \"kubernetes.io/projected/4d43f997-ab21-4268-ade1-9407c9c2e22a-kube-api-access-hbts6\") pod \"payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w\" (UID: \"4d43f997-ab21-4268-ade1-9407c9c2e22a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"
Oct 30 04:31:15.467055 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.467043    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75248e8-cb72-4866-bf44-e9fd5f0b0167-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg\" (UID: \"c75248e8-cb72-4866-bf44-e9fd5f0b0167\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"
Oct 30 04:31:15.467633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.467611    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d43f997-ab21-4268-ade1-9407c9c2e22a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w\" (UID: \"4d43f997-ab21-4268-ade1-9407c9c2e22a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"
Oct 30 04:31:15.491135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.491044    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbts6\" (UniqueName: \"kubernetes.io/projected/4d43f997-ab21-4268-ade1-9407c9c2e22a-kube-api-access-hbts6\") pod \"payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w\" (UID: \"4d43f997-ab21-4268-ade1-9407c9c2e22a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"
Oct 30 04:31:15.547932 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.547911    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"
Oct 30 04:31:15.568192 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.568168    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75248e8-cb72-4866-bf44-e9fd5f0b0167-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg\" (UID: \"c75248e8-cb72-4866-bf44-e9fd5f0b0167\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"
Oct 30 04:31:15.569000 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.568938    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfx2\" (UniqueName: \"kubernetes.io/projected/c75248e8-cb72-4866-bf44-e9fd5f0b0167-kube-api-access-ckfx2\") pod \"payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg\" (UID: \"c75248e8-cb72-4866-bf44-e9fd5f0b0167\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"
Oct 30 04:31:15.569801 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.569254    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75248e8-cb72-4866-bf44-e9fd5f0b0167-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg\" (UID: \"c75248e8-cb72-4866-bf44-e9fd5f0b0167\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"
Oct 30 04:31:15.593471 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.593443    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckfx2\" (UniqueName: \"kubernetes.io/projected/c75248e8-cb72-4866-bf44-e9fd5f0b0167-kube-api-access-ckfx2\") pod \"payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg\" (UID: \"c75248e8-cb72-4866-bf44-e9fd5f0b0167\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"
Oct 30 04:31:15.713786 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.713752    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"
Oct 30 04:31:15.854816 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:15.854789    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"]
Oct 30 04:31:15.854933 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:31:15.854832    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d43f997_ab21_4268_ade1_9407c9c2e22a.slice/crio-7e76a7a4e1b670960905092e1b8bbd001021ab6e61b84fbcb2da066b3de8548b WatchSource:0}: Error finding container 7e76a7a4e1b670960905092e1b8bbd001021ab6e61b84fbcb2da066b3de8548b: Status 404 returned error can't find the container with id 7e76a7a4e1b670960905092e1b8bbd001021ab6e61b84fbcb2da066b3de8548b
Oct 30 04:31:16.025064 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:16.025040    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"]
Oct 30 04:31:16.026299 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:31:16.026270    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75248e8_cb72_4866_bf44_e9fd5f0b0167.slice/crio-ad4b899856a9819e7d88546489d5bac4d5b38226b92531877e8371943cde957d WatchSource:0}: Error finding container ad4b899856a9819e7d88546489d5bac4d5b38226b92531877e8371943cde957d: Status 404 returned error can't find the container with id ad4b899856a9819e7d88546489d5bac4d5b38226b92531877e8371943cde957d
Oct 30 04:31:16.746143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:16.746070    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w" event={"ID":"4d43f997-ab21-4268-ade1-9407c9c2e22a","Type":"ContainerStarted","Data":"9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d"}
Oct 30 04:31:16.746612 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:16.746179    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w" event={"ID":"4d43f997-ab21-4268-ade1-9407c9c2e22a","Type":"ContainerStarted","Data":"7e76a7a4e1b670960905092e1b8bbd001021ab6e61b84fbcb2da066b3de8548b"}
Oct 30 04:31:16.746612 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:16.746221    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"
Oct 30 04:31:16.748092 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:16.748057    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg" event={"ID":"c75248e8-cb72-4866-bf44-e9fd5f0b0167","Type":"ContainerStarted","Data":"e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858"}
Oct 30 04:31:16.748217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:16.748118    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg" event={"ID":"c75248e8-cb72-4866-bf44-e9fd5f0b0167","Type":"ContainerStarted","Data":"ad4b899856a9819e7d88546489d5bac4d5b38226b92531877e8371943cde957d"}
Oct 30 04:31:16.748217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:16.748147    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"
Oct 30 04:31:16.790920 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:16.790875    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w" podStartSLOduration=1.790861958 podStartE2EDuration="1.790861958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:16.78720657 +0000 UTC m=+2056.528629662" watchObservedRunningTime="2026-10-30 04:31:16.790861958 +0000 UTC m=+2056.532285056"
Oct 30 04:31:16.833471 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:16.833422    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg" podStartSLOduration=1.833406928 podStartE2EDuration="1.833406928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:16.830928334 +0000 UTC m=+2056.572351433" watchObservedRunningTime="2026-10-30 04:31:16.833406928 +0000 UTC m=+2056.574830025"
Oct 30 04:31:23.903397 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:23.903366    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"]
Oct 30 04:31:27.767978 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:27.767946    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"
Oct 30 04:31:27.768376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:27.768012    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"
Oct 30 04:31:28.904212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:28.904133    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2" podUID="94c893d7-e177-4f52-b332-b17ce1cc8d9d" containerName="payload-processing" containerID="cri-o://bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3" gracePeriod=25
Oct 30 04:31:29.201868 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.201846    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"
Oct 30 04:31:29.235609 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.235584    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c893d7-e177-4f52-b332-b17ce1cc8d9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c893d7-e177-4f52-b332-b17ce1cc8d9d-plugins-config-volume\") pod \"94c893d7-e177-4f52-b332-b17ce1cc8d9d\" (UID: \"94c893d7-e177-4f52-b332-b17ce1cc8d9d\") "
Oct 30 04:31:29.235737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.235676    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c893d7-e177-4f52-b332-b17ce1cc8d9d-kube-api-access-hc5k6\" (UniqueName: \"kubernetes.io/projected/94c893d7-e177-4f52-b332-b17ce1cc8d9d-kube-api-access-hc5k6\") pod \"94c893d7-e177-4f52-b332-b17ce1cc8d9d\" (UID: \"94c893d7-e177-4f52-b332-b17ce1cc8d9d\") "
Oct 30 04:31:29.235941 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.235921    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c893d7-e177-4f52-b332-b17ce1cc8d9d-plugins-config-volume" pod "94c893d7-e177-4f52-b332-b17ce1cc8d9d" (UID: "94c893d7-e177-4f52-b332-b17ce1cc8d9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:31:29.236140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.236127    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c893d7-e177-4f52-b332-b17ce1cc8d9d-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:31:29.237814 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.237797    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c893d7-e177-4f52-b332-b17ce1cc8d9d-kube-api-access-hc5k6" pod "94c893d7-e177-4f52-b332-b17ce1cc8d9d" (UID: "94c893d7-e177-4f52-b332-b17ce1cc8d9d"). InnerVolumeSpecName "kube-api-access-hc5k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:31:29.337042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.337011    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc5k6\" (UniqueName: \"kubernetes.io/projected/94c893d7-e177-4f52-b332-b17ce1cc8d9d-kube-api-access-hc5k6\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:31:29.851977 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.851943    5072 generic.go:356] "Generic (PLEG): container finished" podID="94c893d7-e177-4f52-b332-b17ce1cc8d9d" containerID="bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3" exitCode=0
Oct 30 04:31:29.852201 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.851995    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2" event={"ID":"94c893d7-e177-4f52-b332-b17ce1cc8d9d","Type":"ContainerDied","Data":"bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3"}
Oct 30 04:31:29.852201 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.852008    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"
Oct 30 04:31:29.852201 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.852112    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2" event={"ID":"94c893d7-e177-4f52-b332-b17ce1cc8d9d","Type":"ContainerDied","Data":"bd2fb2f5fe443ac362935afc057c4b0a22bc3224f8ae6eec27e6e86cb9d58e7d"}
Oct 30 04:31:29.852201 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.852145    5072 scope.go:122] "RemoveContainer" containerID="bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3"
Oct 30 04:31:29.863150 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.863127    5072 scope.go:122] "RemoveContainer" containerID="bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3"
Oct 30 04:31:29.863491 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:31:29.863468    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3\": container with ID starting with bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3 not found: ID does not exist" containerID="bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3"
Oct 30 04:31:29.863577 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.863498    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3"} err="failed to get container status \"bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3\": rpc error: code = NotFound desc = could not find container \"bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3\": container with ID starting with bc28a0e9d49f4257b1fa47c68a6733f34850f7d889fe333cdecfbac5c3c5f0a3 not found: ID does not exist"
Oct 30 04:31:29.905665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.905642    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"]
Oct 30 04:31:29.909928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:29.909905    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a777c-59779bdb8d-hdqk2"]
Oct 30 04:31:30.408232 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:31:30.408201    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c893d7-e177-4f52-b332-b17ce1cc8d9d" path="/var/lib/kubelet/pods/94c893d7-e177-4f52-b332-b17ce1cc8d9d/volumes"
Oct 30 04:31:41.752802 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:31:41.752759    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:32:00.672957 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:00.672929    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:32:00.681779 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:00.681754    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:32:11.751340 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:32:11.751251    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:32:13.702829 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:13.702758    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"]
Oct 30 04:32:13.738267 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:13.738230    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"]
Oct 30 04:32:14.663114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:14.663055    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"]
Oct 30 04:32:14.705427 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:14.705391    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"]
Oct 30 04:32:18.703550 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:18.703494    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh" podUID="b234059b-84a5-423f-b33e-592ca68d9fe0" containerName="payload-processing" containerID="cri-o://578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2" gracePeriod=25
Oct 30 04:32:18.739496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:18.739429    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx" podUID="1645d371-14e9-4c69-be15-cfd189696baf" containerName="payload-pre-processing" containerID="cri-o://cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4" gracePeriod=25
Oct 30 04:32:19.034326 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.034298    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"
Oct 30 04:32:19.081513 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.081483    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"
Oct 30 04:32:19.161611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.137371    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b234059b-84a5-423f-b33e-592ca68d9fe0-kube-api-access-l5b7v\" (UniqueName: \"kubernetes.io/projected/b234059b-84a5-423f-b33e-592ca68d9fe0-kube-api-access-l5b7v\") pod \"b234059b-84a5-423f-b33e-592ca68d9fe0\" (UID: \"b234059b-84a5-423f-b33e-592ca68d9fe0\") "
Oct 30 04:32:19.161611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.137552    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b234059b-84a5-423f-b33e-592ca68d9fe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b234059b-84a5-423f-b33e-592ca68d9fe0-plugins-config-volume\") pod \"b234059b-84a5-423f-b33e-592ca68d9fe0\" (UID: \"b234059b-84a5-423f-b33e-592ca68d9fe0\") "
Oct 30 04:32:19.161611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.141409    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b234059b-84a5-423f-b33e-592ca68d9fe0-plugins-config-volume" pod "b234059b-84a5-423f-b33e-592ca68d9fe0" (UID: "b234059b-84a5-423f-b33e-592ca68d9fe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:32:19.161611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.144812    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b234059b-84a5-423f-b33e-592ca68d9fe0-kube-api-access-l5b7v" pod "b234059b-84a5-423f-b33e-592ca68d9fe0" (UID: "b234059b-84a5-423f-b33e-592ca68d9fe0"). InnerVolumeSpecName "kube-api-access-l5b7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:32:19.168176 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.168139    5072 generic.go:356] "Generic (PLEG): container finished" podID="1645d371-14e9-4c69-be15-cfd189696baf" containerID="cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4" exitCode=0
Oct 30 04:32:19.168342 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.168187    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"
Oct 30 04:32:19.168415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.168364    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx" event={"ID":"1645d371-14e9-4c69-be15-cfd189696baf","Type":"ContainerDied","Data":"cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4"}
Oct 30 04:32:19.168415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.168406    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx" event={"ID":"1645d371-14e9-4c69-be15-cfd189696baf","Type":"ContainerDied","Data":"b6e15a433bac13c38ff6e357446f25b8c3cd8b98c914c168f12b53a3abf7824d"}
Oct 30 04:32:19.168533 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.168432    5072 scope.go:122] "RemoveContainer" containerID="cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4"
Oct 30 04:32:19.170852 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.170827    5072 generic.go:356] "Generic (PLEG): container finished" podID="b234059b-84a5-423f-b33e-592ca68d9fe0" containerID="578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2" exitCode=0
Oct 30 04:32:19.171043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.170875    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh" event={"ID":"b234059b-84a5-423f-b33e-592ca68d9fe0","Type":"ContainerDied","Data":"578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2"}
Oct 30 04:32:19.171043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.170921    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh" event={"ID":"b234059b-84a5-423f-b33e-592ca68d9fe0","Type":"ContainerDied","Data":"ceed4ddfa31c10b752f2996634b1bc1f9a3b2b033cccbbda58449570da90cadf"}
Oct 30 04:32:19.171043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.171003    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"
Oct 30 04:32:19.181343 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.181319    5072 scope.go:122] "RemoveContainer" containerID="cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4"
Oct 30 04:32:19.181700 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:32:19.181679    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4\": container with ID starting with cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4 not found: ID does not exist" containerID="cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4"
Oct 30 04:32:19.181780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.181710    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4"} err="failed to get container status \"cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4\": rpc error: code = NotFound desc = could not find container \"cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4\": container with ID starting with cd4bc3a7897aea624062306633a63dab23ddfdb6fdbc820144596cfe6a3d09c4 not found: ID does not exist"
Oct 30 04:32:19.181780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.181731    5072 scope.go:122] "RemoveContainer" containerID="578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2"
Oct 30 04:32:19.192484 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.192469    5072 scope.go:122] "RemoveContainer" containerID="578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2"
Oct 30 04:32:19.192739 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:32:19.192720    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2\": container with ID starting with 578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2 not found: ID does not exist" containerID="578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2"
Oct 30 04:32:19.192800 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.192748    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2"} err="failed to get container status \"578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2\": rpc error: code = NotFound desc = could not find container \"578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2\": container with ID starting with 578ceee9a89a3c0396e5a7b42921e9fff0ab4a5bfa6c07548d8b0849ed1df0a2 not found: ID does not exist"
Oct 30 04:32:19.222854 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.222831    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"]
Oct 30 04:32:19.228540 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.228512    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-eb9d23-59f4b7d89b-wpphh"]
Oct 30 04:32:19.238716 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.238657    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1645d371-14e9-4c69-be15-cfd189696baf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1645d371-14e9-4c69-be15-cfd189696baf-plugins-config-volume\") pod \"1645d371-14e9-4c69-be15-cfd189696baf\" (UID: \"1645d371-14e9-4c69-be15-cfd189696baf\") "
Oct 30 04:32:19.238814 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.238735    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1645d371-14e9-4c69-be15-cfd189696baf-kube-api-access-7gdrt\" (UniqueName: \"kubernetes.io/projected/1645d371-14e9-4c69-be15-cfd189696baf-kube-api-access-7gdrt\") pod \"1645d371-14e9-4c69-be15-cfd189696baf\" (UID: \"1645d371-14e9-4c69-be15-cfd189696baf\") "
Oct 30 04:32:19.239109 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.239057    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1645d371-14e9-4c69-be15-cfd189696baf-plugins-config-volume" pod "1645d371-14e9-4c69-be15-cfd189696baf" (UID: "1645d371-14e9-4c69-be15-cfd189696baf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:32:19.239215 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.239199    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5b7v\" (UniqueName: \"kubernetes.io/projected/b234059b-84a5-423f-b33e-592ca68d9fe0-kube-api-access-l5b7v\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:32:19.239264 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.239221    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b234059b-84a5-423f-b33e-592ca68d9fe0-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:32:19.239264 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.239237    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1645d371-14e9-4c69-be15-cfd189696baf-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:32:19.240993 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.240973    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1645d371-14e9-4c69-be15-cfd189696baf-kube-api-access-7gdrt" pod "1645d371-14e9-4c69-be15-cfd189696baf" (UID: "1645d371-14e9-4c69-be15-cfd189696baf"). InnerVolumeSpecName "kube-api-access-7gdrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:32:19.340349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.340301    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gdrt\" (UniqueName: \"kubernetes.io/projected/1645d371-14e9-4c69-be15-cfd189696baf-kube-api-access-7gdrt\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:32:19.515960 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.515879    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"]
Oct 30 04:32:19.520502 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.520480    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-eb9d23-59d8b54bc9-2kfrx"]
Oct 30 04:32:19.664404 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.664344    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5" podUID="e7f883b1-9cb3-4f99-bc58-99541ce74e1e" containerName="payload-processing" containerID="cri-o://b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60" gracePeriod=25
Oct 30 04:32:19.706334 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.706300    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh" podUID="dd801aac-9f06-4f1f-b5a4-093f42890475" containerName="payload-pre-processing" containerID="cri-o://882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad" gracePeriod=25
Oct 30 04:32:19.990703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:19.990669    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"
Oct 30 04:32:20.025202 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.025177    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"
Oct 30 04:32:20.047888 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.047858    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-plugins-config-volume\") pod \"e7f883b1-9cb3-4f99-bc58-99541ce74e1e\" (UID: \"e7f883b1-9cb3-4f99-bc58-99541ce74e1e\") "
Oct 30 04:32:20.048274 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.048251    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-plugins-config-volume" pod "e7f883b1-9cb3-4f99-bc58-99541ce74e1e" (UID: "e7f883b1-9cb3-4f99-bc58-99541ce74e1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:32:20.048380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.048362    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:32:20.149541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.149464    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-kube-api-access-jzgvf\" (UniqueName: \"kubernetes.io/projected/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-kube-api-access-jzgvf\") pod \"e7f883b1-9cb3-4f99-bc58-99541ce74e1e\" (UID: \"e7f883b1-9cb3-4f99-bc58-99541ce74e1e\") "
Oct 30 04:32:20.149541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.149501    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd801aac-9f06-4f1f-b5a4-093f42890475-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd801aac-9f06-4f1f-b5a4-093f42890475-plugins-config-volume\") pod \"dd801aac-9f06-4f1f-b5a4-093f42890475\" (UID: \"dd801aac-9f06-4f1f-b5a4-093f42890475\") "
Oct 30 04:32:20.149726 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.149583    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd801aac-9f06-4f1f-b5a4-093f42890475-kube-api-access-6ts94\" (UniqueName: \"kubernetes.io/projected/dd801aac-9f06-4f1f-b5a4-093f42890475-kube-api-access-6ts94\") pod \"dd801aac-9f06-4f1f-b5a4-093f42890475\" (UID: \"dd801aac-9f06-4f1f-b5a4-093f42890475\") "
Oct 30 04:32:20.149899 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.149872    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd801aac-9f06-4f1f-b5a4-093f42890475-plugins-config-volume" pod "dd801aac-9f06-4f1f-b5a4-093f42890475" (UID: "dd801aac-9f06-4f1f-b5a4-093f42890475"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:32:20.151728 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.151708    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-kube-api-access-jzgvf" pod "e7f883b1-9cb3-4f99-bc58-99541ce74e1e" (UID: "e7f883b1-9cb3-4f99-bc58-99541ce74e1e"). InnerVolumeSpecName "kube-api-access-jzgvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:32:20.151803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.151766    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd801aac-9f06-4f1f-b5a4-093f42890475-kube-api-access-6ts94" pod "dd801aac-9f06-4f1f-b5a4-093f42890475" (UID: "dd801aac-9f06-4f1f-b5a4-093f42890475"). InnerVolumeSpecName "kube-api-access-6ts94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:32:20.179002 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.178976    5072 generic.go:356] "Generic (PLEG): container finished" podID="dd801aac-9f06-4f1f-b5a4-093f42890475" containerID="882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad" exitCode=0
Oct 30 04:32:20.179155 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.179023    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"
Oct 30 04:32:20.179155 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.179061    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh" event={"ID":"dd801aac-9f06-4f1f-b5a4-093f42890475","Type":"ContainerDied","Data":"882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad"}
Oct 30 04:32:20.179155 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.179143    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh" event={"ID":"dd801aac-9f06-4f1f-b5a4-093f42890475","Type":"ContainerDied","Data":"ac7d335d60376b18f22aa6bc2b6ea50f241e8c28a901dededecc6d2c0ba2dfd9"}
Oct 30 04:32:20.179293 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.179179    5072 scope.go:122] "RemoveContainer" containerID="882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad"
Oct 30 04:32:20.181689 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.181664    5072 generic.go:356] "Generic (PLEG): container finished" podID="e7f883b1-9cb3-4f99-bc58-99541ce74e1e" containerID="b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60" exitCode=0
Oct 30 04:32:20.181796 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.181702    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"
Oct 30 04:32:20.181796 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.181752    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5" event={"ID":"e7f883b1-9cb3-4f99-bc58-99541ce74e1e","Type":"ContainerDied","Data":"b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60"}
Oct 30 04:32:20.181796 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.181791    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5" event={"ID":"e7f883b1-9cb3-4f99-bc58-99541ce74e1e","Type":"ContainerDied","Data":"c6c663e1253ec9daf4acf5ffecf71964da632c371eed126d4d3d966b09f25c9d"}
Oct 30 04:32:20.191559 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.191544    5072 scope.go:122] "RemoveContainer" containerID="882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad"
Oct 30 04:32:20.191787 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:32:20.191766    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad\": container with ID starting with 882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad not found: ID does not exist" containerID="882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad"
Oct 30 04:32:20.191850 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.191794    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad"} err="failed to get container status \"882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad\": rpc error: code = NotFound desc = could not find container \"882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad\": container with ID starting with 882b8cf7cc1594e0e2ac54ee04962397027d4b3097ef622f7f7c946045ebdaad not found: ID does not exist"
Oct 30 04:32:20.191850 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.191810    5072 scope.go:122] "RemoveContainer" containerID="b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60"
Oct 30 04:32:20.200771 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.200752    5072 scope.go:122] "RemoveContainer" containerID="b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60"
Oct 30 04:32:20.201021 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:32:20.201005    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60\": container with ID starting with b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60 not found: ID does not exist" containerID="b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60"
Oct 30 04:32:20.201061 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.201027    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60"} err="failed to get container status \"b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60\": rpc error: code = NotFound desc = could not find container \"b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60\": container with ID starting with b10371489e6e20bbbf363c7a1978af1e7bace95c1e128d112636ff7d5b33ea60 not found: ID does not exist"
Oct 30 04:32:20.234499 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.234477    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"]
Oct 30 04:32:20.239265 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.239244    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-05f827-58cdf7cf99-bbdgh"]
Oct 30 04:32:20.250583 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.250563    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ts94\" (UniqueName: \"kubernetes.io/projected/dd801aac-9f06-4f1f-b5a4-093f42890475-kube-api-access-6ts94\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:32:20.250669 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.250583    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzgvf\" (UniqueName: \"kubernetes.io/projected/e7f883b1-9cb3-4f99-bc58-99541ce74e1e-kube-api-access-jzgvf\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:32:20.250669 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.250626    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd801aac-9f06-4f1f-b5a4-093f42890475-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:32:20.274471 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.274431    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"]
Oct 30 04:32:20.278001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.277980    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-05f827-68c47d899d-2zzk5"]
Oct 30 04:32:20.408272 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.408205    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1645d371-14e9-4c69-be15-cfd189696baf" path="/var/lib/kubelet/pods/1645d371-14e9-4c69-be15-cfd189696baf/volumes"
Oct 30 04:32:20.408621 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.408608    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b234059b-84a5-423f-b33e-592ca68d9fe0" path="/var/lib/kubelet/pods/b234059b-84a5-423f-b33e-592ca68d9fe0/volumes"
Oct 30 04:32:20.408945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.408933    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd801aac-9f06-4f1f-b5a4-093f42890475" path="/var/lib/kubelet/pods/dd801aac-9f06-4f1f-b5a4-093f42890475/volumes"
Oct 30 04:32:20.409294 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:20.409281    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f883b1-9cb3-4f99-bc58-99541ce74e1e" path="/var/lib/kubelet/pods/e7f883b1-9cb3-4f99-bc58-99541ce74e1e/volumes"
Oct 30 04:32:41.751189 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:32:41.751151    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:32:45.805327 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.805288    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"]
Oct 30 04:32:45.805903 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.805888    5072 cpu_manager.go:414] "containerMap: removing container" podUID="b234059b-84a5-423f-b33e-592ca68d9fe0" containerName="payload-processing"
Oct 30 04:32:45.805949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.805906    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b234059b-84a5-423f-b33e-592ca68d9fe0" containerName="payload-processing"
Oct 30 04:32:45.805949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.805930    5072 cpu_manager.go:414] "containerMap: removing container" podUID="1645d371-14e9-4c69-be15-cfd189696baf" containerName="payload-pre-processing"
Oct 30 04:32:45.805949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.805936    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1645d371-14e9-4c69-be15-cfd189696baf" containerName="payload-pre-processing"
Oct 30 04:32:45.806046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.805951    5072 cpu_manager.go:414] "containerMap: removing container" podUID="dd801aac-9f06-4f1f-b5a4-093f42890475" containerName="payload-pre-processing"
Oct 30 04:32:45.806046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.805955    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd801aac-9f06-4f1f-b5a4-093f42890475" containerName="payload-pre-processing"
Oct 30 04:32:45.806046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.805963    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e7f883b1-9cb3-4f99-bc58-99541ce74e1e" containerName="payload-processing"
Oct 30 04:32:45.806046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.805967    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f883b1-9cb3-4f99-bc58-99541ce74e1e" containerName="payload-processing"
Oct 30 04:32:45.806046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.805991    5072 cpu_manager.go:414] "containerMap: removing container" podUID="94c893d7-e177-4f52-b332-b17ce1cc8d9d" containerName="payload-processing"
Oct 30 04:32:45.806046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.805998    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c893d7-e177-4f52-b332-b17ce1cc8d9d" containerName="payload-processing"
Oct 30 04:32:45.806258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.806098    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f883b1-9cb3-4f99-bc58-99541ce74e1e" containerName="payload-processing"
Oct 30 04:32:45.806258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.806116    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c893d7-e177-4f52-b332-b17ce1cc8d9d" containerName="payload-processing"
Oct 30 04:32:45.806258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.806125    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b234059b-84a5-423f-b33e-592ca68d9fe0" containerName="payload-processing"
Oct 30 04:32:45.806258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.806133    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1645d371-14e9-4c69-be15-cfd189696baf" containerName="payload-pre-processing"
Oct 30 04:32:45.806258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.806141    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd801aac-9f06-4f1f-b5a4-093f42890475" containerName="payload-pre-processing"
Oct 30 04:32:45.809367 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.809342    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:45.827364 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.827337    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"]
Oct 30 04:32:45.836803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.836784    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:32:45.836916 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.836908    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:32:45.967500 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.967466    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxtsz\" (UniqueName: \"kubernetes.io/projected/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-kube-api-access-fxtsz\") pod \"maas-api-revoke-sub-b941509b-9tvfc\" (UID: \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:45.967674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.967538    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b941509b-9tvfc\" (UID: \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:45.967719 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:45.967673    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b941509b-9tvfc\" (UID: \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:46.068444 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:46.068359    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b941509b-9tvfc\" (UID: \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:46.068599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:46.068452    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxtsz\" (UniqueName: \"kubernetes.io/projected/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-kube-api-access-fxtsz\") pod \"maas-api-revoke-sub-b941509b-9tvfc\" (UID: \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:46.068599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:46.068505    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b941509b-9tvfc\" (UID: \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:46.069229 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:46.069171    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b941509b-9tvfc\" (UID: \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:46.070864 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:46.070845    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b941509b-9tvfc\" (UID: \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:46.092726 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:46.092704    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxtsz\" (UniqueName: \"kubernetes.io/projected/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-kube-api-access-fxtsz\") pod \"maas-api-revoke-sub-b941509b-9tvfc\" (UID: \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:46.123235 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:46.123209    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:46.435950 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:46.435914    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"]
Oct 30 04:32:46.436976 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:32:46.436947    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af3b23f_4a32_40b4_9f0a_0c77f1675cc1.slice/crio-876dade7590f06e6b262f595c94a4d2f17384f5f723b165dffea8905b21285ad WatchSource:0}: Error finding container 876dade7590f06e6b262f595c94a4d2f17384f5f723b165dffea8905b21285ad: Status 404 returned error can't find the container with id 876dade7590f06e6b262f595c94a4d2f17384f5f723b165dffea8905b21285ad
Oct 30 04:32:47.384714 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:47.384690    5072 generic.go:356] "Generic (PLEG): container finished" podID="3af3b23f-4a32-40b4-9f0a-0c77f1675cc1" containerID="6d2a1b2b6fcd8da2951adedb2432ab1da24012c13e078ec906793c7671f7068f" exitCode=0
Oct 30 04:32:47.385054 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:47.384744    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc" event={"ID":"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1","Type":"ContainerDied","Data":"6d2a1b2b6fcd8da2951adedb2432ab1da24012c13e078ec906793c7671f7068f"}
Oct 30 04:32:47.385054 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:47.384785    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc" event={"ID":"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1","Type":"ContainerStarted","Data":"876dade7590f06e6b262f595c94a4d2f17384f5f723b165dffea8905b21285ad"}
Oct 30 04:32:48.545401 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:48.545376    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:48.696640 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:48.696611    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-kube-api-access-fxtsz\" (UniqueName: \"kubernetes.io/projected/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-kube-api-access-fxtsz\") pod \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\" (UID: \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\") "
Oct 30 04:32:48.696816 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:48.696660    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-xks-ca\") pod \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\" (UID: \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\") "
Oct 30 04:32:48.696816 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:48.696703    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-openshift-ca\") pod \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\" (UID: \"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1\") "
Oct 30 04:32:48.697112 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:48.697086    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-openshift-ca" pod "3af3b23f-4a32-40b4-9f0a-0c77f1675cc1" (UID: "3af3b23f-4a32-40b4-9f0a-0c77f1675cc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:32:48.697259 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:48.697245    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:32:48.698868 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:48.698847    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-xks-ca" pod "3af3b23f-4a32-40b4-9f0a-0c77f1675cc1" (UID: "3af3b23f-4a32-40b4-9f0a-0c77f1675cc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:32:48.698868 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:48.698851    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-kube-api-access-fxtsz" pod "3af3b23f-4a32-40b4-9f0a-0c77f1675cc1" (UID: "3af3b23f-4a32-40b4-9f0a-0c77f1675cc1"). InnerVolumeSpecName "kube-api-access-fxtsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:32:48.798586 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:48.798558    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxtsz\" (UniqueName: \"kubernetes.io/projected/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-kube-api-access-fxtsz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:32:48.798586 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:48.798583    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:32:49.406777 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:49.406748    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"
Oct 30 04:32:49.406981 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:49.406744    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc" event={"ID":"3af3b23f-4a32-40b4-9f0a-0c77f1675cc1","Type":"ContainerDied","Data":"876dade7590f06e6b262f595c94a4d2f17384f5f723b165dffea8905b21285ad"}
Oct 30 04:32:49.406981 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:49.406886    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="876dade7590f06e6b262f595c94a4d2f17384f5f723b165dffea8905b21285ad"
Oct 30 04:32:57.970693 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:57.970651    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"]
Oct 30 04:32:57.971448 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:57.971429    5072 cpu_manager.go:414] "containerMap: removing container" podUID="3af3b23f-4a32-40b4-9f0a-0c77f1675cc1" containerName="revoke-keys"
Oct 30 04:32:57.971574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:57.971450    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af3b23f-4a32-40b4-9f0a-0c77f1675cc1" containerName="revoke-keys"
Oct 30 04:32:57.971661 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:57.971647    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af3b23f-4a32-40b4-9f0a-0c77f1675cc1" containerName="revoke-keys"
Oct 30 04:32:57.974933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:57.974913    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:32:57.992934 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:57.992911    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"]
Oct 30 04:32:57.999365 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:57.999334    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvzpn\" (UniqueName: \"kubernetes.io/projected/92271683-f03f-423a-a74d-7149b9338a75-kube-api-access-fvzpn\") pod \"maas-api-revoke-sub-b5222649-kkvsp\" (UID: \"92271683-f03f-423a-a74d-7149b9338a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:32:57.999499 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:57.999469    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92271683-f03f-423a-a74d-7149b9338a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5222649-kkvsp\" (UID: \"92271683-f03f-423a-a74d-7149b9338a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:32:57.999572 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:57.999518    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92271683-f03f-423a-a74d-7149b9338a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5222649-kkvsp\" (UID: \"92271683-f03f-423a-a74d-7149b9338a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:32:58.002115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:58.002029    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:32:58.002239 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:58.002218    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:32:58.100440 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:58.100406    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvzpn\" (UniqueName: \"kubernetes.io/projected/92271683-f03f-423a-a74d-7149b9338a75-kube-api-access-fvzpn\") pod \"maas-api-revoke-sub-b5222649-kkvsp\" (UID: \"92271683-f03f-423a-a74d-7149b9338a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:32:58.100599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:58.100491    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92271683-f03f-423a-a74d-7149b9338a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5222649-kkvsp\" (UID: \"92271683-f03f-423a-a74d-7149b9338a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:32:58.100599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:58.100531    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92271683-f03f-423a-a74d-7149b9338a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5222649-kkvsp\" (UID: \"92271683-f03f-423a-a74d-7149b9338a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:32:58.101146 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:58.101125    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92271683-f03f-423a-a74d-7149b9338a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5222649-kkvsp\" (UID: \"92271683-f03f-423a-a74d-7149b9338a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:32:58.103025 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:58.103005    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92271683-f03f-423a-a74d-7149b9338a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5222649-kkvsp\" (UID: \"92271683-f03f-423a-a74d-7149b9338a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:32:58.126090 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:58.126047    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvzpn\" (UniqueName: \"kubernetes.io/projected/92271683-f03f-423a-a74d-7149b9338a75-kube-api-access-fvzpn\") pod \"maas-api-revoke-sub-b5222649-kkvsp\" (UID: \"92271683-f03f-423a-a74d-7149b9338a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:32:58.288795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:58.288724    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:32:58.595846 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:58.595820    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"]
Oct 30 04:32:59.471882 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:59.471845    5072 generic.go:356] "Generic (PLEG): container finished" podID="92271683-f03f-423a-a74d-7149b9338a75" containerID="bbb8c88a278275b2259e8d65949b7d7570da1a374db4cddbd71be1b69a11f825" exitCode=0
Oct 30 04:32:59.472313 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:59.471926    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp" event={"ID":"92271683-f03f-423a-a74d-7149b9338a75","Type":"ContainerDied","Data":"bbb8c88a278275b2259e8d65949b7d7570da1a374db4cddbd71be1b69a11f825"}
Oct 30 04:32:59.472313 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:32:59.472001    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp" event={"ID":"92271683-f03f-423a-a74d-7149b9338a75","Type":"ContainerStarted","Data":"26cf9e6c86b57f389093af8f45a266359b25c82d67467ec20289424773db53b2"}
Oct 30 04:33:00.727727 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:00.727703    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:33:00.835725 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:00.835650    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92271683-f03f-423a-a74d-7149b9338a75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92271683-f03f-423a-a74d-7149b9338a75-maas-api-openshift-ca\") pod \"92271683-f03f-423a-a74d-7149b9338a75\" (UID: \"92271683-f03f-423a-a74d-7149b9338a75\") "
Oct 30 04:33:00.835725 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:00.835713    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92271683-f03f-423a-a74d-7149b9338a75-kube-api-access-fvzpn\" (UniqueName: \"kubernetes.io/projected/92271683-f03f-423a-a74d-7149b9338a75-kube-api-access-fvzpn\") pod \"92271683-f03f-423a-a74d-7149b9338a75\" (UID: \"92271683-f03f-423a-a74d-7149b9338a75\") "
Oct 30 04:33:00.835953 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:00.835739    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92271683-f03f-423a-a74d-7149b9338a75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92271683-f03f-423a-a74d-7149b9338a75-maas-api-xks-ca\") pod \"92271683-f03f-423a-a74d-7149b9338a75\" (UID: \"92271683-f03f-423a-a74d-7149b9338a75\") "
Oct 30 04:33:00.836017 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:00.835982    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92271683-f03f-423a-a74d-7149b9338a75-maas-api-openshift-ca" pod "92271683-f03f-423a-a74d-7149b9338a75" (UID: "92271683-f03f-423a-a74d-7149b9338a75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:33:00.836168 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:00.836152    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92271683-f03f-423a-a74d-7149b9338a75-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:33:00.838030 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:00.838000    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92271683-f03f-423a-a74d-7149b9338a75-maas-api-xks-ca" pod "92271683-f03f-423a-a74d-7149b9338a75" (UID: "92271683-f03f-423a-a74d-7149b9338a75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:33:00.838198 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:00.838045    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92271683-f03f-423a-a74d-7149b9338a75-kube-api-access-fvzpn" pod "92271683-f03f-423a-a74d-7149b9338a75" (UID: "92271683-f03f-423a-a74d-7149b9338a75"). InnerVolumeSpecName "kube-api-access-fvzpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:33:00.937142 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:00.937117    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvzpn\" (UniqueName: \"kubernetes.io/projected/92271683-f03f-423a-a74d-7149b9338a75-kube-api-access-fvzpn\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:33:00.937142 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:00.937142    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92271683-f03f-423a-a74d-7149b9338a75-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:33:01.487775 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:01.487747    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"
Oct 30 04:33:01.487954 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:01.487741    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp" event={"ID":"92271683-f03f-423a-a74d-7149b9338a75","Type":"ContainerDied","Data":"26cf9e6c86b57f389093af8f45a266359b25c82d67467ec20289424773db53b2"}
Oct 30 04:33:01.487954 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:01.487872    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26cf9e6c86b57f389093af8f45a266359b25c82d67467ec20289424773db53b2"
Oct 30 04:33:10.032960 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:10.032928    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"]
Oct 30 04:33:10.045390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:10.045362    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6aa08f-kzqtt"]
Oct 30 04:33:10.409657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:10.409575    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf49544-93d5-40fa-9f12-5d56e15610d9" path="/var/lib/kubelet/pods/8cf49544-93d5-40fa-9f12-5d56e15610d9/volumes"
Oct 30 04:33:11.016523 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.016480    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"]
Oct 30 04:33:11.017369 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.017344    5072 cpu_manager.go:414] "containerMap: removing container" podUID="92271683-f03f-423a-a74d-7149b9338a75" containerName="revoke-keys"
Oct 30 04:33:11.017538 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.017525    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92271683-f03f-423a-a74d-7149b9338a75" containerName="revoke-keys"
Oct 30 04:33:11.017826 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.017813    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="92271683-f03f-423a-a74d-7149b9338a75" containerName="revoke-keys"
Oct 30 04:33:11.024004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.023970    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:11.044791 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.044762    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"]
Oct 30 04:33:11.070398 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.070368    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:33:11.071557 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.070637    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:33:11.172923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.172883    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-954991a2-ftqqx\" (UID: \"432f5fa3-94d9-4559-95c1-f260a0570e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:11.173149 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.172950    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-954991a2-ftqqx\" (UID: \"432f5fa3-94d9-4559-95c1-f260a0570e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:11.173149 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.173016    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9tmh\" (UniqueName: \"kubernetes.io/projected/432f5fa3-94d9-4559-95c1-f260a0570e8f-kube-api-access-l9tmh\") pod \"maas-api-revoke-sub-954991a2-ftqqx\" (UID: \"432f5fa3-94d9-4559-95c1-f260a0570e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:11.274349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.274250    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-954991a2-ftqqx\" (UID: \"432f5fa3-94d9-4559-95c1-f260a0570e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:11.274349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.274316    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-954991a2-ftqqx\" (UID: \"432f5fa3-94d9-4559-95c1-f260a0570e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:11.274598 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.274383    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9tmh\" (UniqueName: \"kubernetes.io/projected/432f5fa3-94d9-4559-95c1-f260a0570e8f-kube-api-access-l9tmh\") pod \"maas-api-revoke-sub-954991a2-ftqqx\" (UID: \"432f5fa3-94d9-4559-95c1-f260a0570e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:11.275615 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.275572    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-954991a2-ftqqx\" (UID: \"432f5fa3-94d9-4559-95c1-f260a0570e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:11.278114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.278012    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-954991a2-ftqqx\" (UID: \"432f5fa3-94d9-4559-95c1-f260a0570e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:11.304498 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.304464    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9tmh\" (UniqueName: \"kubernetes.io/projected/432f5fa3-94d9-4559-95c1-f260a0570e8f-kube-api-access-l9tmh\") pod \"maas-api-revoke-sub-954991a2-ftqqx\" (UID: \"432f5fa3-94d9-4559-95c1-f260a0570e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:11.351630 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.351593    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:11.706185 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:11.700205    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"]
Oct 30 04:33:11.762141 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:33:11.762039    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:33:12.572596 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:12.572554    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx" event={"ID":"432f5fa3-94d9-4559-95c1-f260a0570e8f","Type":"ContainerStarted","Data":"c1100e13daedc801867a8218757275263b6ef4b2cc9aef71342878f6ce613379"}
Oct 30 04:33:12.572992 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:12.572649    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx" event={"ID":"432f5fa3-94d9-4559-95c1-f260a0570e8f","Type":"ContainerStarted","Data":"ace30129ced4bed2a7bbe21b973e82daeedc3dbb078dde636626f1b0dda4da91"}
Oct 30 04:33:12.615973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:12.615926    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx" podStartSLOduration=2.615913474 podStartE2EDuration="2.615913474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:12.613346368 +0000 UTC m=+2172.354769466" watchObservedRunningTime="2026-10-30 04:33:12.615913474 +0000 UTC m=+2172.357336571"
Oct 30 04:33:13.579974 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:13.579938    5072 generic.go:356] "Generic (PLEG): container finished" podID="432f5fa3-94d9-4559-95c1-f260a0570e8f" containerID="c1100e13daedc801867a8218757275263b6ef4b2cc9aef71342878f6ce613379" exitCode=0
Oct 30 04:33:13.580406 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:13.579980    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx" event={"ID":"432f5fa3-94d9-4559-95c1-f260a0570e8f","Type":"ContainerDied","Data":"c1100e13daedc801867a8218757275263b6ef4b2cc9aef71342878f6ce613379"}
Oct 30 04:33:14.805092 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:14.805045    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:14.926705 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:14.926634    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-xks-ca\") pod \"432f5fa3-94d9-4559-95c1-f260a0570e8f\" (UID: \"432f5fa3-94d9-4559-95c1-f260a0570e8f\") "
Oct 30 04:33:14.926855 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:14.926724    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432f5fa3-94d9-4559-95c1-f260a0570e8f-kube-api-access-l9tmh\" (UniqueName: \"kubernetes.io/projected/432f5fa3-94d9-4559-95c1-f260a0570e8f-kube-api-access-l9tmh\") pod \"432f5fa3-94d9-4559-95c1-f260a0570e8f\" (UID: \"432f5fa3-94d9-4559-95c1-f260a0570e8f\") "
Oct 30 04:33:14.926855 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:14.926766    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-openshift-ca\") pod \"432f5fa3-94d9-4559-95c1-f260a0570e8f\" (UID: \"432f5fa3-94d9-4559-95c1-f260a0570e8f\") "
Oct 30 04:33:14.927196 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:14.927164    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-openshift-ca" pod "432f5fa3-94d9-4559-95c1-f260a0570e8f" (UID: "432f5fa3-94d9-4559-95c1-f260a0570e8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:33:14.929066 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:14.929032    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-xks-ca" pod "432f5fa3-94d9-4559-95c1-f260a0570e8f" (UID: "432f5fa3-94d9-4559-95c1-f260a0570e8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:33:14.929066 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:14.929053    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432f5fa3-94d9-4559-95c1-f260a0570e8f-kube-api-access-l9tmh" pod "432f5fa3-94d9-4559-95c1-f260a0570e8f" (UID: "432f5fa3-94d9-4559-95c1-f260a0570e8f"). InnerVolumeSpecName "kube-api-access-l9tmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:33:15.028716 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:15.028685    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:33:15.028716 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:15.028713    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9tmh\" (UniqueName: \"kubernetes.io/projected/432f5fa3-94d9-4559-95c1-f260a0570e8f-kube-api-access-l9tmh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:33:15.028716 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:15.028722    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432f5fa3-94d9-4559-95c1-f260a0570e8f-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:33:15.592764 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:15.592733    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"
Oct 30 04:33:15.592764 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:15.592754    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx" event={"ID":"432f5fa3-94d9-4559-95c1-f260a0570e8f","Type":"ContainerDied","Data":"ace30129ced4bed2a7bbe21b973e82daeedc3dbb078dde636626f1b0dda4da91"}
Oct 30 04:33:15.592972 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:15.592800    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ace30129ced4bed2a7bbe21b973e82daeedc3dbb078dde636626f1b0dda4da91"
Oct 30 04:33:20.134762 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.134722    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"]
Oct 30 04:33:20.135602 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.135580    5072 cpu_manager.go:414] "containerMap: removing container" podUID="432f5fa3-94d9-4559-95c1-f260a0570e8f" containerName="revoke-keys"
Oct 30 04:33:20.135703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.135602    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432f5fa3-94d9-4559-95c1-f260a0570e8f" containerName="revoke-keys"
Oct 30 04:33:20.135760 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.135750    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="432f5fa3-94d9-4559-95c1-f260a0570e8f" containerName="revoke-keys"
Oct 30 04:33:20.140446 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.140424    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:20.157710 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.157686    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"]
Oct 30 04:33:20.166647 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.166610    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:33:20.166813 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.166665    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:33:20.187576 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.187549    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d7n2\" (UniqueName: \"kubernetes.io/projected/02973297-549f-4801-9c83-d5553fc1e00f-kube-api-access-5d7n2\") pod \"maas-api-revoke-sub-86ad5966-ffrdm\" (UID: \"02973297-549f-4801-9c83-d5553fc1e00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:20.187720 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.187592    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ad5966-ffrdm\" (UID: \"02973297-549f-4801-9c83-d5553fc1e00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:20.187770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.187750    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ad5966-ffrdm\" (UID: \"02973297-549f-4801-9c83-d5553fc1e00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:20.289038 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.288987    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d7n2\" (UniqueName: \"kubernetes.io/projected/02973297-549f-4801-9c83-d5553fc1e00f-kube-api-access-5d7n2\") pod \"maas-api-revoke-sub-86ad5966-ffrdm\" (UID: \"02973297-549f-4801-9c83-d5553fc1e00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:20.289217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.289051    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ad5966-ffrdm\" (UID: \"02973297-549f-4801-9c83-d5553fc1e00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:20.289217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.289175    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ad5966-ffrdm\" (UID: \"02973297-549f-4801-9c83-d5553fc1e00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:20.289747 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.289716    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ad5966-ffrdm\" (UID: \"02973297-549f-4801-9c83-d5553fc1e00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:20.291938 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.291918    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ad5966-ffrdm\" (UID: \"02973297-549f-4801-9c83-d5553fc1e00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:20.312770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.312752    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d7n2\" (UniqueName: \"kubernetes.io/projected/02973297-549f-4801-9c83-d5553fc1e00f-kube-api-access-5d7n2\") pod \"maas-api-revoke-sub-86ad5966-ffrdm\" (UID: \"02973297-549f-4801-9c83-d5553fc1e00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:20.454665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.454637    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:20.757667 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:20.757641    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"]
Oct 30 04:33:20.758212 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:33:20.758183    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02973297_549f_4801_9c83_d5553fc1e00f.slice/crio-442b1fc1f95d0d1d57ed9f89d8ba695d68c22d9530ec8971196b95c8081fe723 WatchSource:0}: Error finding container 442b1fc1f95d0d1d57ed9f89d8ba695d68c22d9530ec8971196b95c8081fe723: Status 404 returned error can't find the container with id 442b1fc1f95d0d1d57ed9f89d8ba695d68c22d9530ec8971196b95c8081fe723
Oct 30 04:33:21.634070 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:21.634037    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm" event={"ID":"02973297-549f-4801-9c83-d5553fc1e00f","Type":"ContainerStarted","Data":"463a01d9827b99f1cdea57defe029b5a1de96fe27e9338705b710314f2dd00ae"}
Oct 30 04:33:21.634522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:21.634105    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm" event={"ID":"02973297-549f-4801-9c83-d5553fc1e00f","Type":"ContainerStarted","Data":"442b1fc1f95d0d1d57ed9f89d8ba695d68c22d9530ec8971196b95c8081fe723"}
Oct 30 04:33:21.677960 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:21.677916    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm" podStartSLOduration=1.6779014330000002 podStartE2EDuration="1.677901433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:21.67513985 +0000 UTC m=+2181.416562947" watchObservedRunningTime="2026-10-30 04:33:21.677901433 +0000 UTC m=+2181.419324556"
Oct 30 04:33:22.641330 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:22.641294    5072 generic.go:356] "Generic (PLEG): container finished" podID="02973297-549f-4801-9c83-d5553fc1e00f" containerID="463a01d9827b99f1cdea57defe029b5a1de96fe27e9338705b710314f2dd00ae" exitCode=0
Oct 30 04:33:22.641728 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:22.641386    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm" event={"ID":"02973297-549f-4801-9c83-d5553fc1e00f","Type":"ContainerDied","Data":"463a01d9827b99f1cdea57defe029b5a1de96fe27e9338705b710314f2dd00ae"}
Oct 30 04:33:23.960063 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:23.960035    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:24.040430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:24.040401    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-xks-ca\") pod \"02973297-549f-4801-9c83-d5553fc1e00f\" (UID: \"02973297-549f-4801-9c83-d5553fc1e00f\") "
Oct 30 04:33:24.040621 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:24.040495    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-openshift-ca\") pod \"02973297-549f-4801-9c83-d5553fc1e00f\" (UID: \"02973297-549f-4801-9c83-d5553fc1e00f\") "
Oct 30 04:33:24.040621 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:24.040533    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02973297-549f-4801-9c83-d5553fc1e00f-kube-api-access-5d7n2\" (UniqueName: \"kubernetes.io/projected/02973297-549f-4801-9c83-d5553fc1e00f-kube-api-access-5d7n2\") pod \"02973297-549f-4801-9c83-d5553fc1e00f\" (UID: \"02973297-549f-4801-9c83-d5553fc1e00f\") "
Oct 30 04:33:24.040924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:24.040893    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-openshift-ca" pod "02973297-549f-4801-9c83-d5553fc1e00f" (UID: "02973297-549f-4801-9c83-d5553fc1e00f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:33:24.041187 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:24.041162    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:33:24.042835 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:24.042807    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-xks-ca" pod "02973297-549f-4801-9c83-d5553fc1e00f" (UID: "02973297-549f-4801-9c83-d5553fc1e00f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:33:24.043195 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:24.043176    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02973297-549f-4801-9c83-d5553fc1e00f-kube-api-access-5d7n2" pod "02973297-549f-4801-9c83-d5553fc1e00f" (UID: "02973297-549f-4801-9c83-d5553fc1e00f"). InnerVolumeSpecName "kube-api-access-5d7n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:33:24.143133 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:24.143100    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02973297-549f-4801-9c83-d5553fc1e00f-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:33:24.143289 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:24.143138    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d7n2\" (UniqueName: \"kubernetes.io/projected/02973297-549f-4801-9c83-d5553fc1e00f-kube-api-access-5d7n2\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:33:24.660819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:24.660792    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"
Oct 30 04:33:24.661024 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:24.660816    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm" event={"ID":"02973297-549f-4801-9c83-d5553fc1e00f","Type":"ContainerDied","Data":"442b1fc1f95d0d1d57ed9f89d8ba695d68c22d9530ec8971196b95c8081fe723"}
Oct 30 04:33:24.661024 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:33:24.660869    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="442b1fc1f95d0d1d57ed9f89d8ba695d68c22d9530ec8971196b95c8081fe723"
Oct 30 04:33:41.752331 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:33:41.752248    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:34:11.754801 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:34:11.754766    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:34:29.404376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:29.404336    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"]
Oct 30 04:34:29.457671 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:29.457632    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"]
Oct 30 04:34:34.405156 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.405094    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg" podUID="c75248e8-cb72-4866-bf44-e9fd5f0b0167" containerName="payload-processing" containerID="cri-o://e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858" gracePeriod=25
Oct 30 04:34:34.458547 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.458495    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w" podUID="4d43f997-ab21-4268-ade1-9407c9c2e22a" containerName="payload-pre-processing" containerID="cri-o://9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d" gracePeriod=25
Oct 30 04:34:34.729213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.729185    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"
Oct 30 04:34:34.774770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.774738    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"
Oct 30 04:34:34.803283 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.803249    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75248e8-cb72-4866-bf44-e9fd5f0b0167-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75248e8-cb72-4866-bf44-e9fd5f0b0167-plugins-config-volume\") pod \"c75248e8-cb72-4866-bf44-e9fd5f0b0167\" (UID: \"c75248e8-cb72-4866-bf44-e9fd5f0b0167\") "
Oct 30 04:34:34.803491 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.803430    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75248e8-cb72-4866-bf44-e9fd5f0b0167-kube-api-access-ckfx2\" (UniqueName: \"kubernetes.io/projected/c75248e8-cb72-4866-bf44-e9fd5f0b0167-kube-api-access-ckfx2\") pod \"c75248e8-cb72-4866-bf44-e9fd5f0b0167\" (UID: \"c75248e8-cb72-4866-bf44-e9fd5f0b0167\") "
Oct 30 04:34:34.803743 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.803716    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75248e8-cb72-4866-bf44-e9fd5f0b0167-plugins-config-volume" pod "c75248e8-cb72-4866-bf44-e9fd5f0b0167" (UID: "c75248e8-cb72-4866-bf44-e9fd5f0b0167"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:34:34.805726 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.805701    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75248e8-cb72-4866-bf44-e9fd5f0b0167-kube-api-access-ckfx2" pod "c75248e8-cb72-4866-bf44-e9fd5f0b0167" (UID: "c75248e8-cb72-4866-bf44-e9fd5f0b0167"). InnerVolumeSpecName "kube-api-access-ckfx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:34.904192 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.904157    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d43f997-ab21-4268-ade1-9407c9c2e22a-kube-api-access-hbts6\" (UniqueName: \"kubernetes.io/projected/4d43f997-ab21-4268-ade1-9407c9c2e22a-kube-api-access-hbts6\") pod \"4d43f997-ab21-4268-ade1-9407c9c2e22a\" (UID: \"4d43f997-ab21-4268-ade1-9407c9c2e22a\") "
Oct 30 04:34:34.904372 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.904211    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d43f997-ab21-4268-ade1-9407c9c2e22a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d43f997-ab21-4268-ade1-9407c9c2e22a-plugins-config-volume\") pod \"4d43f997-ab21-4268-ade1-9407c9c2e22a\" (UID: \"4d43f997-ab21-4268-ade1-9407c9c2e22a\") "
Oct 30 04:34:34.904596 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.904571    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckfx2\" (UniqueName: \"kubernetes.io/projected/c75248e8-cb72-4866-bf44-e9fd5f0b0167-kube-api-access-ckfx2\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:34:34.904596 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.904591    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75248e8-cb72-4866-bf44-e9fd5f0b0167-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:34:34.904685 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.904638    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d43f997-ab21-4268-ade1-9407c9c2e22a-plugins-config-volume" pod "4d43f997-ab21-4268-ade1-9407c9c2e22a" (UID: "4d43f997-ab21-4268-ade1-9407c9c2e22a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:34:34.906588 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:34.906564    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d43f997-ab21-4268-ade1-9407c9c2e22a-kube-api-access-hbts6" pod "4d43f997-ab21-4268-ade1-9407c9c2e22a" (UID: "4d43f997-ab21-4268-ade1-9407c9c2e22a"). InnerVolumeSpecName "kube-api-access-hbts6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:35.006380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.006300    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbts6\" (UniqueName: \"kubernetes.io/projected/4d43f997-ab21-4268-ade1-9407c9c2e22a-kube-api-access-hbts6\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:34:35.006380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.006328    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d43f997-ab21-4268-ade1-9407c9c2e22a-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:34:35.140973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.140939    5072 generic.go:356] "Generic (PLEG): container finished" podID="c75248e8-cb72-4866-bf44-e9fd5f0b0167" containerID="e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858" exitCode=0
Oct 30 04:34:35.141196 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.140997    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"
Oct 30 04:34:35.141196 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.141028    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg" event={"ID":"c75248e8-cb72-4866-bf44-e9fd5f0b0167","Type":"ContainerDied","Data":"e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858"}
Oct 30 04:34:35.141196 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.141126    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg" event={"ID":"c75248e8-cb72-4866-bf44-e9fd5f0b0167","Type":"ContainerDied","Data":"ad4b899856a9819e7d88546489d5bac4d5b38226b92531877e8371943cde957d"}
Oct 30 04:34:35.141196 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.141156    5072 scope.go:122] "RemoveContainer" containerID="e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858"
Oct 30 04:34:35.142534 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.142515    5072 generic.go:356] "Generic (PLEG): container finished" podID="4d43f997-ab21-4268-ade1-9407c9c2e22a" containerID="9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d" exitCode=0
Oct 30 04:34:35.142651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.142573    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"
Oct 30 04:34:35.142651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.142584    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w" event={"ID":"4d43f997-ab21-4268-ade1-9407c9c2e22a","Type":"ContainerDied","Data":"9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d"}
Oct 30 04:34:35.142651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.142615    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w" event={"ID":"4d43f997-ab21-4268-ade1-9407c9c2e22a","Type":"ContainerDied","Data":"7e76a7a4e1b670960905092e1b8bbd001021ab6e61b84fbcb2da066b3de8548b"}
Oct 30 04:34:35.152745 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.152719    5072 scope.go:122] "RemoveContainer" containerID="e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858"
Oct 30 04:34:35.153132 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:34:35.153110    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858\": container with ID starting with e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858 not found: ID does not exist" containerID="e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858"
Oct 30 04:34:35.153218 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.153141    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858"} err="failed to get container status \"e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858\": rpc error: code = NotFound desc = could not find container \"e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858\": container with ID starting with e84387f792c48ad7993d5006c74d33432493222641a20010e922fc9a26742858 not found: ID does not exist"
Oct 30 04:34:35.153218 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.153159    5072 scope.go:122] "RemoveContainer" containerID="9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d"
Oct 30 04:34:35.163585 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.163562    5072 scope.go:122] "RemoveContainer" containerID="9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d"
Oct 30 04:34:35.163879 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:34:35.163858    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d\": container with ID starting with 9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d not found: ID does not exist" containerID="9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d"
Oct 30 04:34:35.163953 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.163892    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d"} err="failed to get container status \"9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d\": rpc error: code = NotFound desc = could not find container \"9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d\": container with ID starting with 9de5d6cc368125e2c9a5dbe7fd1f63190313ef12fd42bed3dd455b85c70cb86d not found: ID does not exist"
Oct 30 04:34:35.194848 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.194815    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"]
Oct 30 04:34:35.198441 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.198417    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-448a3a-f449fbb48-cdqxg"]
Oct 30 04:34:35.233021 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.232985    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"]
Oct 30 04:34:35.236955 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:35.236923    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-448a3a-559597875-cxb2w"]
Oct 30 04:34:36.411290 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:36.411251    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d43f997-ab21-4268-ade1-9407c9c2e22a" path="/var/lib/kubelet/pods/4d43f997-ab21-4268-ade1-9407c9c2e22a/volumes"
Oct 30 04:34:36.411995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:34:36.411970    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75248e8-cb72-4866-bf44-e9fd5f0b0167" path="/var/lib/kubelet/pods/c75248e8-cb72-4866-bf44-e9fd5f0b0167/volumes"
Oct 30 04:34:41.753687 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:34:41.753645    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:35:00.933281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:35:00.933243    5072 scope.go:122] "RemoveContainer" containerID="423874d3683691442230f845607aa9f44e9dc4dcee9a8653fac2e0dc4a6bfeb2"
Oct 30 04:35:11.754348 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:35:11.754255    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:35:41.750579 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:35:41.750540    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:36:11.755327 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:36:11.755282    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:36:41.754052 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:36:41.753974    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:37:00.713994 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:00.713968    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:37:00.724333 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:00.724311    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:37:11.752155 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:37:11.752115    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:37:22.456370 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:22.456334    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"]
Oct 30 04:37:22.456830 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:22.456568    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" podUID="090baa36-2795-4057-9963-5f9a0e448b55" containerName="istio-proxy" containerID="cri-o://ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca" gracePeriod=30
Oct 30 04:37:27.729350 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.729287    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:37:27.872173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872142    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-certs\") pod \"090baa36-2795-4057-9963-5f9a0e448b55\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") "
Oct 30 04:37:27.872341 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872189    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-credential-socket\") pod \"090baa36-2795-4057-9963-5f9a0e448b55\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") "
Oct 30 04:37:27.872341 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872223    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/090baa36-2795-4057-9963-5f9a0e448b55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090baa36-2795-4057-9963-5f9a0e448b55-istio-podinfo\") pod \"090baa36-2795-4057-9963-5f9a0e448b55\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") "
Oct 30 04:37:27.872341 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872281    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-envoy\") pod \"090baa36-2795-4057-9963-5f9a0e448b55\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") "
Oct 30 04:37:27.872341 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872330    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-socket\") pod \"090baa36-2795-4057-9963-5f9a0e448b55\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") "
Oct 30 04:37:27.872570 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872393    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-data\") pod \"090baa36-2795-4057-9963-5f9a0e448b55\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") "
Oct 30 04:37:27.872570 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872460    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-istio-token\" (UniqueName: \"kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-istio-token\") pod \"090baa36-2795-4057-9963-5f9a0e448b55\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") "
Oct 30 04:37:27.872570 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872469    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-credential-socket" pod "090baa36-2795-4057-9963-5f9a0e448b55" (UID: "090baa36-2795-4057-9963-5f9a0e448b55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:37:27.872570 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872457    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-certs" pod "090baa36-2795-4057-9963-5f9a0e448b55" (UID: "090baa36-2795-4057-9963-5f9a0e448b55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:37:27.872570 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872539    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/090baa36-2795-4057-9963-5f9a0e448b55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090baa36-2795-4057-9963-5f9a0e448b55-istiod-ca-cert\") pod \"090baa36-2795-4057-9963-5f9a0e448b55\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") "
Oct 30 04:37:27.872826 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872585    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-kube-api-access-xjb9p\" (UniqueName: \"kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-kube-api-access-xjb9p\") pod \"090baa36-2795-4057-9963-5f9a0e448b55\" (UID: \"090baa36-2795-4057-9963-5f9a0e448b55\") "
Oct 30 04:37:27.872826 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872780    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-data" pod "090baa36-2795-4057-9963-5f9a0e448b55" (UID: "090baa36-2795-4057-9963-5f9a0e448b55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:37:27.872999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.872973    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090baa36-2795-4057-9963-5f9a0e448b55-istiod-ca-cert" pod "090baa36-2795-4057-9963-5f9a0e448b55" (UID: "090baa36-2795-4057-9963-5f9a0e448b55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:37:27.873326 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.873069    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-socket" pod "090baa36-2795-4057-9963-5f9a0e448b55" (UID: "090baa36-2795-4057-9963-5f9a0e448b55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:37:27.873326 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.873180    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:27.873326 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.873203    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:27.873326 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.873221    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:27.873326 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.873234    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:27.873326 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.873247    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090baa36-2795-4057-9963-5f9a0e448b55-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:27.874748 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.874720    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/090baa36-2795-4057-9963-5f9a0e448b55-istio-podinfo" pod "090baa36-2795-4057-9963-5f9a0e448b55" (UID: "090baa36-2795-4057-9963-5f9a0e448b55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:37:27.874861 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.874761    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-istio-token" pod "090baa36-2795-4057-9963-5f9a0e448b55" (UID: "090baa36-2795-4057-9963-5f9a0e448b55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:37:27.874926 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.874912    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-envoy" pod "090baa36-2795-4057-9963-5f9a0e448b55" (UID: "090baa36-2795-4057-9963-5f9a0e448b55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:37:27.874987 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.874972    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-kube-api-access-xjb9p" pod "090baa36-2795-4057-9963-5f9a0e448b55" (UID: "090baa36-2795-4057-9963-5f9a0e448b55"). InnerVolumeSpecName "kube-api-access-xjb9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:37:27.974518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.974487    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090baa36-2795-4057-9963-5f9a0e448b55-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:27.974518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.974511    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:27.974518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.974521    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjb9p\" (UniqueName: \"kubernetes.io/projected/090baa36-2795-4057-9963-5f9a0e448b55-kube-api-access-xjb9p\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:27.974754 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:27.974530    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090baa36-2795-4057-9963-5f9a0e448b55-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:28.238485 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:28.238451    5072 generic.go:356] "Generic (PLEG): container finished" podID="090baa36-2795-4057-9963-5f9a0e448b55" containerID="ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca" exitCode=0
Oct 30 04:37:28.238663 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:28.238512    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"
Oct 30 04:37:28.238663 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:28.238535    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" event={"ID":"090baa36-2795-4057-9963-5f9a0e448b55","Type":"ContainerDied","Data":"ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca"}
Oct 30 04:37:28.238663 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:28.238596    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms" event={"ID":"090baa36-2795-4057-9963-5f9a0e448b55","Type":"ContainerDied","Data":"c2cb1e5fd05f053cfb15929c8e54bfca5b746b97d95ac542d1151a6f8e33f272"}
Oct 30 04:37:28.238663 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:28.238618    5072 scope.go:122] "RemoveContainer" containerID="ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca"
Oct 30 04:37:28.253748 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:28.253725    5072 scope.go:122] "RemoveContainer" containerID="ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca"
Oct 30 04:37:28.254229 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:37:28.254209    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca\": container with ID starting with ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca not found: ID does not exist" containerID="ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca"
Oct 30 04:37:28.254321 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:28.254240    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca"} err="failed to get container status \"ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca\": rpc error: code = NotFound desc = could not find container \"ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca\": container with ID starting with ce4283ce7e272f2ce96ca93dc1bb0b2f107b1d0c21228eba8dd6cc6829713cca not found: ID does not exist"
Oct 30 04:37:28.293554 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:28.293522    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"]
Oct 30 04:37:28.306434 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:28.306410    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sw4ms"]
Oct 30 04:37:28.408715 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:28.408672    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090baa36-2795-4057-9963-5f9a0e448b55" path="/var/lib/kubelet/pods/090baa36-2795-4057-9963-5f9a0e448b55/volumes"
Oct 30 04:37:41.751581 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:37:41.751538    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:37:45.575488 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.575432    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"]
Oct 30 04:37:45.575993 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.575698    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr" podUID="e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" containerName="discovery" containerID="cri-o://1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3" gracePeriod=30
Oct 30 04:37:45.843758 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.843705    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:37:45.883656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.883633    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-cacerts\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-cacerts\") pod \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") "
Oct 30 04:37:45.883805 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.883670    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-kube-api-access-5cvt5\" (UniqueName: \"kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-kube-api-access-5cvt5\") pod \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") "
Oct 30 04:37:45.883805 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.883773    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-token\" (UniqueName: \"kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-token\") pod \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") "
Oct 30 04:37:45.883936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.883837    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-local-certs\") pod \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") "
Oct 30 04:37:45.883936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.883877    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-ca-configmap\") pod \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") "
Oct 30 04:37:45.883936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.883923    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-dns-cert\") pod \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") "
Oct 30 04:37:45.884131 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.883959    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-kubeconfig\") pod \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\" (UID: \"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc\") "
Oct 30 04:37:45.884688 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.884360    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-ca-configmap" pod "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" (UID: "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:37:45.886562 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.886535    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-kube-api-access-5cvt5" pod "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" (UID: "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc"). InnerVolumeSpecName "kube-api-access-5cvt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:37:45.886784 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.886742    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-cacerts" pod "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" (UID: "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:37:45.887154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.887126    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-token" pod "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" (UID: "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:37:45.887256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.887190    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-kubeconfig" pod "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" (UID: "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:37:45.887256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.887239    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-dns-cert" pod "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" (UID: "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:37:45.887543 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.887271    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-local-certs" pod "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" (UID: "e9172cb2-60dd-4c80-8704-b5e7e82e5fdc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:37:45.985045 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.985016    5072 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-local-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:45.985045 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.985045    5072 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-ca-configmap\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:45.985269 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.985059    5072 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-csr-dns-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:45.985269 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.985071    5072 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-kubeconfig\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:45.985269 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.985120    5072 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-cacerts\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:45.985269 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.985132    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvt5\" (UniqueName: \"kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-kube-api-access-5cvt5\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:45.985269 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:45.985144    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:37:46.346526 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:46.346494    5072 generic.go:356] "Generic (PLEG): container finished" podID="e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" containerID="1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3" exitCode=0
Oct 30 04:37:46.346704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:46.346539    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"
Oct 30 04:37:46.346704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:46.346581    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr" event={"ID":"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc","Type":"ContainerDied","Data":"1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3"}
Oct 30 04:37:46.346704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:46.346640    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr" event={"ID":"e9172cb2-60dd-4c80-8704-b5e7e82e5fdc","Type":"ContainerDied","Data":"b2b3a7efa3edd2aeb877a4654d775aff10049931ae0256ea8dfc641a15ae9257"}
Oct 30 04:37:46.346704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:46.346661    5072 scope.go:122] "RemoveContainer" containerID="1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3"
Oct 30 04:37:46.357594 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:46.357577    5072 scope.go:122] "RemoveContainer" containerID="1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3"
Oct 30 04:37:46.357860 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:37:46.357840    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3\": container with ID starting with 1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3 not found: ID does not exist" containerID="1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3"
Oct 30 04:37:46.357916 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:46.357871    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3"} err="failed to get container status \"1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3\": rpc error: code = NotFound desc = could not find container \"1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3\": container with ID starting with 1c30e77468b9309c1590fbfad13eaf06e1e07aeeaebcb982d898d0a22d9444d3 not found: ID does not exist"
Oct 30 04:37:46.416314 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:46.416286    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"]
Oct 30 04:37:46.416448 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:46.416331    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j24vr"]
Oct 30 04:37:48.407649 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:48.407620    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" path="/var/lib/kubelet/pods/e9172cb2-60dd-4c80-8704-b5e7e82e5fdc/volumes"
Oct 30 04:37:49.033299 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:49.033267    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"]
Oct 30 04:37:49.046054 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:49.046019    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b941509b-9tvfc"]
Oct 30 04:37:50.409355 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:37:50.409315    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af3b23f-4a32-40b4-9f0a-0c77f1675cc1" path="/var/lib/kubelet/pods/3af3b23f-4a32-40b4-9f0a-0c77f1675cc1/volumes"
Oct 30 04:38:01.029687 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:38:01.029659    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"]
Oct 30 04:38:01.041877 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:38:01.041851    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5222649-kkvsp"]
Oct 30 04:38:02.407763 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:38:02.407727    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92271683-f03f-423a-a74d-7149b9338a75" path="/var/lib/kubelet/pods/92271683-f03f-423a-a74d-7149b9338a75/volumes"
Oct 30 04:38:11.751970 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:38:11.751873    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:38:15.038480 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:38:15.038433    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"]
Oct 30 04:38:15.051654 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:38:15.051623    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-954991a2-ftqqx"]
Oct 30 04:38:16.407714 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:38:16.407682    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432f5fa3-94d9-4559-95c1-f260a0570e8f" path="/var/lib/kubelet/pods/432f5fa3-94d9-4559-95c1-f260a0570e8f/volumes"
Oct 30 04:38:24.034906 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:38:24.034866    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"]
Oct 30 04:38:24.047322 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:38:24.047287    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ad5966-ffrdm"]
Oct 30 04:38:24.407816 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:38:24.407744    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02973297-549f-4801-9c83-d5553fc1e00f" path="/var/lib/kubelet/pods/02973297-549f-4801-9c83-d5553fc1e00f/volumes"
Oct 30 04:38:41.750741 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:38:41.750705    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:39:01.002512 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:01.002479    5072 scope.go:122] "RemoveContainer" containerID="bbb8c88a278275b2259e8d65949b7d7570da1a374db4cddbd71be1b69a11f825"
Oct 30 04:39:01.012284 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:01.012260    5072 scope.go:122] "RemoveContainer" containerID="6d2a1b2b6fcd8da2951adedb2432ab1da24012c13e078ec906793c7671f7068f"
Oct 30 04:39:11.751391 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:39:11.751347    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:39:41.756275 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:39:41.756187    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:39:52.853248 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:52.853215    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"]
Oct 30 04:39:52.853682 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:52.853422    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" podUID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" containerName="istio-proxy" containerID="cri-o://2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498" gracePeriod=30
Oct 30 04:39:58.128171 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.128114    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:39:58.136802 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.136767    5072 generic.go:356] "Generic (PLEG): container finished" podID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" containerID="2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498" exitCode=0
Oct 30 04:39:58.136935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.136818    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"
Oct 30 04:39:58.136935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.136846    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" event={"ID":"b8168654-1ce2-4b2b-b09a-42c08d07b31d","Type":"ContainerDied","Data":"2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498"}
Oct 30 04:39:58.136935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.136891    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr" event={"ID":"b8168654-1ce2-4b2b-b09a-42c08d07b31d","Type":"ContainerDied","Data":"52f8a4c6761facc51edc3a1c1e7c055293b55ffe5159014d0dabc602c9205f7d"}
Oct 30 04:39:58.136935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.136920    5072 scope.go:122] "RemoveContainer" containerID="2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498"
Oct 30 04:39:58.146473 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.146457    5072 scope.go:122] "RemoveContainer" containerID="2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498"
Oct 30 04:39:58.146744 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:39:58.146729    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498\": container with ID starting with 2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498 not found: ID does not exist" containerID="2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498"
Oct 30 04:39:58.146786 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.146753    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498"} err="failed to get container status \"2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498\": rpc error: code = NotFound desc = could not find container \"2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498\": container with ID starting with 2080f68fd4bafae0e642f75e0770473c0edf1f972a0eb4cbb61d23fc875bc498 not found: ID does not exist"
Oct 30 04:39:58.263140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263109    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-socket\") pod \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") "
Oct 30 04:39:58.263296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263207    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-envoy\") pod \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") "
Oct 30 04:39:58.263296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263236    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-token\" (UniqueName: \"kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-token\") pod \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") "
Oct 30 04:39:58.263296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263282    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-data\") pod \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") "
Oct 30 04:39:58.263453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263330    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-credential-socket\") pod \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") "
Oct 30 04:39:58.263453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263370    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-kube-api-access-xkgmx\" (UniqueName: \"kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-kube-api-access-xkgmx\") pod \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") "
Oct 30 04:39:58.263453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263403    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-socket" pod "b8168654-1ce2-4b2b-b09a-42c08d07b31d" (UID: "b8168654-1ce2-4b2b-b09a-42c08d07b31d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:58.263453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263408    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-certs\") pod \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") "
Oct 30 04:39:58.263646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263482    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-podinfo\") pod \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") "
Oct 30 04:39:58.263646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263548    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istiod-ca-cert\") pod \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\" (UID: \"b8168654-1ce2-4b2b-b09a-42c08d07b31d\") "
Oct 30 04:39:58.263646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263617    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-data" pod "b8168654-1ce2-4b2b-b09a-42c08d07b31d" (UID: "b8168654-1ce2-4b2b-b09a-42c08d07b31d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:58.263646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263632    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-credential-socket" pod "b8168654-1ce2-4b2b-b09a-42c08d07b31d" (UID: "b8168654-1ce2-4b2b-b09a-42c08d07b31d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:58.263840 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263649    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-certs" pod "b8168654-1ce2-4b2b-b09a-42c08d07b31d" (UID: "b8168654-1ce2-4b2b-b09a-42c08d07b31d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:58.264023 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.263979    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:39:58.264023 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.264009    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:39:58.264167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.264031    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:39:58.264167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.264046    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:39:58.264167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.264004    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istiod-ca-cert" pod "b8168654-1ce2-4b2b-b09a-42c08d07b31d" (UID: "b8168654-1ce2-4b2b-b09a-42c08d07b31d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:39:58.265606 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.265581    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-token" pod "b8168654-1ce2-4b2b-b09a-42c08d07b31d" (UID: "b8168654-1ce2-4b2b-b09a-42c08d07b31d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:39:58.265732 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.265662    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-envoy" pod "b8168654-1ce2-4b2b-b09a-42c08d07b31d" (UID: "b8168654-1ce2-4b2b-b09a-42c08d07b31d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:58.265799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.265730    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-podinfo" pod "b8168654-1ce2-4b2b-b09a-42c08d07b31d" (UID: "b8168654-1ce2-4b2b-b09a-42c08d07b31d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:39:58.266010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.265995    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-kube-api-access-xkgmx" pod "b8168654-1ce2-4b2b-b09a-42c08d07b31d" (UID: "b8168654-1ce2-4b2b-b09a-42c08d07b31d"). InnerVolumeSpecName "kube-api-access-xkgmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:39:58.365376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.365347    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:39:58.365376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.365371    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:39:58.365376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.365380    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:39:58.365583 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.365389    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkgmx\" (UniqueName: \"kubernetes.io/projected/b8168654-1ce2-4b2b-b09a-42c08d07b31d-kube-api-access-xkgmx\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:39:58.365583 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.365398    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8168654-1ce2-4b2b-b09a-42c08d07b31d-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:39:58.478973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.478936    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"]
Oct 30 04:39:58.485525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:39:58.485498    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-987vr"]
Oct 30 04:40:00.408314 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:00.408281    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" path="/var/lib/kubelet/pods/b8168654-1ce2-4b2b-b09a-42c08d07b31d/volumes"
Oct 30 04:40:01.038675 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:01.038645    5072 scope.go:122] "RemoveContainer" containerID="c1100e13daedc801867a8218757275263b6ef4b2cc9aef71342878f6ce613379"
Oct 30 04:40:01.047854 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:01.047829    5072 scope.go:122] "RemoveContainer" containerID="463a01d9827b99f1cdea57defe029b5a1de96fe27e9338705b710314f2dd00ae"
Oct 30 04:40:11.754115 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:40:11.754041    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:40:15.923883 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.923851    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"]
Oct 30 04:40:15.924524 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924507    5072 cpu_manager.go:414] "containerMap: removing container" podUID="4d43f997-ab21-4268-ade1-9407c9c2e22a" containerName="payload-pre-processing"
Oct 30 04:40:15.924577 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924526    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d43f997-ab21-4268-ade1-9407c9c2e22a" containerName="payload-pre-processing"
Oct 30 04:40:15.924577 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924557    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" containerName="discovery"
Oct 30 04:40:15.924577 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924562    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" containerName="discovery"
Oct 30 04:40:15.924577 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924574    5072 cpu_manager.go:414] "containerMap: removing container" podUID="090baa36-2795-4057-9963-5f9a0e448b55" containerName="istio-proxy"
Oct 30 04:40:15.924713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924580    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090baa36-2795-4057-9963-5f9a0e448b55" containerName="istio-proxy"
Oct 30 04:40:15.924713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924590    5072 cpu_manager.go:414] "containerMap: removing container" podUID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" containerName="istio-proxy"
Oct 30 04:40:15.924713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924595    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" containerName="istio-proxy"
Oct 30 04:40:15.924713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924604    5072 cpu_manager.go:414] "containerMap: removing container" podUID="c75248e8-cb72-4866-bf44-e9fd5f0b0167" containerName="payload-processing"
Oct 30 04:40:15.924713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924608    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75248e8-cb72-4866-bf44-e9fd5f0b0167" containerName="payload-processing"
Oct 30 04:40:15.924713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924618    5072 cpu_manager.go:414] "containerMap: removing container" podUID="02973297-549f-4801-9c83-d5553fc1e00f" containerName="revoke-keys"
Oct 30 04:40:15.924713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924641    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02973297-549f-4801-9c83-d5553fc1e00f" containerName="revoke-keys"
Oct 30 04:40:15.924918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924722    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75248e8-cb72-4866-bf44-e9fd5f0b0167" containerName="payload-processing"
Oct 30 04:40:15.924918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924730    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="02973297-549f-4801-9c83-d5553fc1e00f" containerName="revoke-keys"
Oct 30 04:40:15.924918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924738    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d43f997-ab21-4268-ade1-9407c9c2e22a" containerName="payload-pre-processing"
Oct 30 04:40:15.924918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924749    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9172cb2-60dd-4c80-8704-b5e7e82e5fdc" containerName="discovery"
Oct 30 04:40:15.924918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924755    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="090baa36-2795-4057-9963-5f9a0e448b55" containerName="istio-proxy"
Oct 30 04:40:15.924918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.924760    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8168654-1ce2-4b2b-b09a-42c08d07b31d" containerName="istio-proxy"
Oct 30 04:40:15.929755 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.929736    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"
Oct 30 04:40:15.945256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.945191    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"]
Oct 30 04:40:15.955239 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.955207    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-495ebe\""
Oct 30 04:40:15.955352 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:15.955267    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-495ebe-dockercfg-nnwmd\""
Oct 30 04:40:16.001668 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.001632    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"]
Oct 30 04:40:16.015045 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.015016    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"
Oct 30 04:40:16.038118 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.033528    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"]
Oct 30 04:40:16.064124 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.055811    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d1ddbc-3156-44bf-be1d-9f236418a2b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl\" (UID: \"35d1ddbc-3156-44bf-be1d-9f236418a2b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"
Oct 30 04:40:16.064124 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.061190    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v8sv\" (UniqueName: \"kubernetes.io/projected/35d1ddbc-3156-44bf-be1d-9f236418a2b7-kube-api-access-4v8sv\") pod \"payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl\" (UID: \"35d1ddbc-3156-44bf-be1d-9f236418a2b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"
Oct 30 04:40:16.162430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.162400    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v8sv\" (UniqueName: \"kubernetes.io/projected/35d1ddbc-3156-44bf-be1d-9f236418a2b7-kube-api-access-4v8sv\") pod \"payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl\" (UID: \"35d1ddbc-3156-44bf-be1d-9f236418a2b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"
Oct 30 04:40:16.162678 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.162503    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m\" (UID: \"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"
Oct 30 04:40:16.162678 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.162647    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c48s5\" (UniqueName: \"kubernetes.io/projected/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-kube-api-access-c48s5\") pod \"payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m\" (UID: \"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"
Oct 30 04:40:16.162771 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.162683    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d1ddbc-3156-44bf-be1d-9f236418a2b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl\" (UID: \"35d1ddbc-3156-44bf-be1d-9f236418a2b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"
Oct 30 04:40:16.163348 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.163330    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d1ddbc-3156-44bf-be1d-9f236418a2b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl\" (UID: \"35d1ddbc-3156-44bf-be1d-9f236418a2b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"
Oct 30 04:40:16.186712 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.186683    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v8sv\" (UniqueName: \"kubernetes.io/projected/35d1ddbc-3156-44bf-be1d-9f236418a2b7-kube-api-access-4v8sv\") pod \"payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl\" (UID: \"35d1ddbc-3156-44bf-be1d-9f236418a2b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"
Oct 30 04:40:16.244049 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.244019    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"
Oct 30 04:40:16.263951 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.263921    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c48s5\" (UniqueName: \"kubernetes.io/projected/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-kube-api-access-c48s5\") pod \"payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m\" (UID: \"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"
Oct 30 04:40:16.264108 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.264046    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m\" (UID: \"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"
Oct 30 04:40:16.264970 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.264941    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m\" (UID: \"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"
Oct 30 04:40:16.288353 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.288324    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c48s5\" (UniqueName: \"kubernetes.io/projected/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-kube-api-access-c48s5\") pod \"payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m\" (UID: \"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"
Oct 30 04:40:16.350697 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.350660    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"
Oct 30 04:40:16.588356 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:40:16.588323    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d1ddbc_3156_44bf_be1d_9f236418a2b7.slice/crio-8637ff64eb8630c98d6f10d2a4003228e80c6ee05eabb3aeb0c7168c68d11337 WatchSource:0}: Error finding container 8637ff64eb8630c98d6f10d2a4003228e80c6ee05eabb3aeb0c7168c68d11337: Status 404 returned error can't find the container with id 8637ff64eb8630c98d6f10d2a4003228e80c6ee05eabb3aeb0c7168c68d11337
Oct 30 04:40:16.590395 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.590363    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"]
Oct 30 04:40:16.676914 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:16.676887    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"]
Oct 30 04:40:16.677741 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:40:16.677704    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc260ccd2_e290_4e56_84f0_2ebc79a5fcf7.slice/crio-5d802c64b464357086c0125dd2b6fe958f2f95e3055b53c3ced58cbe0dc45d32 WatchSource:0}: Error finding container 5d802c64b464357086c0125dd2b6fe958f2f95e3055b53c3ced58cbe0dc45d32: Status 404 returned error can't find the container with id 5d802c64b464357086c0125dd2b6fe958f2f95e3055b53c3ced58cbe0dc45d32
Oct 30 04:40:17.249570 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.249527    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl" event={"ID":"35d1ddbc-3156-44bf-be1d-9f236418a2b7","Type":"ContainerStarted","Data":"b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb"}
Oct 30 04:40:17.250002 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.249594    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"
Oct 30 04:40:17.250002 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.249622    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl" event={"ID":"35d1ddbc-3156-44bf-be1d-9f236418a2b7","Type":"ContainerStarted","Data":"8637ff64eb8630c98d6f10d2a4003228e80c6ee05eabb3aeb0c7168c68d11337"}
Oct 30 04:40:17.251301 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.251274    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m" event={"ID":"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7","Type":"ContainerStarted","Data":"cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9"}
Oct 30 04:40:17.251433 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.251320    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m" event={"ID":"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7","Type":"ContainerStarted","Data":"5d802c64b464357086c0125dd2b6fe958f2f95e3055b53c3ced58cbe0dc45d32"}
Oct 30 04:40:17.251433 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.251377    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"
Oct 30 04:40:17.291299 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.291253    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl" podStartSLOduration=2.291240171 podStartE2EDuration="2.291240171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:17.290946286 +0000 UTC m=+2597.032369384" watchObservedRunningTime="2026-10-30 04:40:17.291240171 +0000 UTC m=+2597.032663268"
Oct 30 04:40:17.604206 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.604100    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m" podStartSLOduration=2.604068826 podStartE2EDuration="2.604068826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:17.335447581 +0000 UTC m=+2597.076870677" watchObservedRunningTime="2026-10-30 04:40:17.604068826 +0000 UTC m=+2597.345491943"
Oct 30 04:40:17.606569 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.606541    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"]
Oct 30 04:40:17.610845 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.610815    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.626734 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.626630    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-fcd6a6\"/\"openshift-service-ca.crt\""
Oct 30 04:40:17.627160 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.626993    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-fcd6a6\"/\"default-dockercfg-xdjqn\""
Oct 30 04:40:17.627351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.627325    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-fcd6a6\"/\"auth-test-model-fcd6a6-kserve-self-signed-certs\""
Oct 30 04:40:17.627465 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.627439    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-fcd6a6\"/\"kube-root-ca.crt\""
Oct 30 04:40:17.628531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.628508    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"]
Oct 30 04:40:17.781449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.781416    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-home\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.781626 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.781477    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6abc515-914d-4443-9237-8b677d1594c1-tls-certs\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.781681 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.781627    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8g58\" (UniqueName: \"kubernetes.io/projected/f6abc515-914d-4443-9237-8b677d1594c1-kube-api-access-x8g58\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.781752 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.781736    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-model-cache\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.781798 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.781771    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-dshm\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.781902 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.781884    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-tmp-dir\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.883121 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.883005    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-home\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.883121 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.883056    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6abc515-914d-4443-9237-8b677d1594c1-tls-certs\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.883323 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.883132    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8g58\" (UniqueName: \"kubernetes.io/projected/f6abc515-914d-4443-9237-8b677d1594c1-kube-api-access-x8g58\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.883323 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.883205    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-model-cache\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.883323 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.883233    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-dshm\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.883323 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.883297    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-tmp-dir\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.883536 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.883511    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-home\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.883592 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.883577    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-model-cache\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.883679 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.883659    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-tmp-dir\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.885672 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.885646    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-dshm\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.885966 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.885949    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6abc515-914d-4443-9237-8b677d1594c1-tls-certs\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.907528 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.907493    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8g58\" (UniqueName: \"kubernetes.io/projected/f6abc515-914d-4443-9237-8b677d1594c1-kube-api-access-x8g58\") pod \"auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") " pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:17.926249 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:17.926224    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:18.228777 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:18.228751    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"]
Oct 30 04:40:18.229160 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:40:18.229135    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6abc515_914d_4443_9237_8b677d1594c1.slice/crio-d51cec534f51252e3395e8b7e88c00f3ec76fbb5a851d8847278b8f3527b1518 WatchSource:0}: Error finding container d51cec534f51252e3395e8b7e88c00f3ec76fbb5a851d8847278b8f3527b1518: Status 404 returned error can't find the container with id d51cec534f51252e3395e8b7e88c00f3ec76fbb5a851d8847278b8f3527b1518
Oct 30 04:40:18.260130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:18.260102    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f" event={"ID":"f6abc515-914d-4443-9237-8b677d1594c1","Type":"ContainerStarted","Data":"d51cec534f51252e3395e8b7e88c00f3ec76fbb5a851d8847278b8f3527b1518"}
Oct 30 04:40:19.267629 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:19.267590    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f" event={"ID":"f6abc515-914d-4443-9237-8b677d1594c1","Type":"ContainerStarted","Data":"8f661381f1fe66bd09853307e9fc0703e91218df72b65d22dd20060dbc90aa02"}
Oct 30 04:40:19.312567 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:19.312520    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f" podStartSLOduration=2.312506469 podStartE2EDuration="2.312506469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:19.312107115 +0000 UTC m=+2599.053530260" watchObservedRunningTime="2026-10-30 04:40:19.312506469 +0000 UTC m=+2599.053929566"
Oct 30 04:40:19.926477 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:19.926447    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:27.926351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:27.926312    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:27.939498 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:27.939472    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:28.263613 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:28.263582    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"
Oct 30 04:40:28.263778 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:28.263656    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"
Oct 30 04:40:28.354426 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:28.354396    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:40:30.472646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:30.472610    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"]
Oct 30 04:40:30.477633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:30.477609    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"
Oct 30 04:40:30.497771 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:30.497749    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"]
Oct 30 04:40:30.506783 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:30.506763    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-247158-dockercfg-6dqtk\""
Oct 30 04:40:30.506783 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:30.506778    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-247158\""
Oct 30 04:40:30.538256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:30.537059    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdqh\" (UniqueName: \"kubernetes.io/projected/ed107f59-f9a9-4a99-8a89-26a288611dd1-kube-api-access-8zdqh\") pod \"payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t\" (UID: \"ed107f59-f9a9-4a99-8a89-26a288611dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"
Oct 30 04:40:30.538256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:30.537232    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed107f59-f9a9-4a99-8a89-26a288611dd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t\" (UID: \"ed107f59-f9a9-4a99-8a89-26a288611dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"
Oct 30 04:40:30.638476 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:30.638431    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdqh\" (UniqueName: \"kubernetes.io/projected/ed107f59-f9a9-4a99-8a89-26a288611dd1-kube-api-access-8zdqh\") pod \"payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t\" (UID: \"ed107f59-f9a9-4a99-8a89-26a288611dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"
Oct 30 04:40:30.638651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:30.638526    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed107f59-f9a9-4a99-8a89-26a288611dd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t\" (UID: \"ed107f59-f9a9-4a99-8a89-26a288611dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"
Oct 30 04:40:30.639571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:30.639539    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed107f59-f9a9-4a99-8a89-26a288611dd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t\" (UID: \"ed107f59-f9a9-4a99-8a89-26a288611dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"
Oct 30 04:40:30.665392 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:30.665363    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdqh\" (UniqueName: \"kubernetes.io/projected/ed107f59-f9a9-4a99-8a89-26a288611dd1-kube-api-access-8zdqh\") pod \"payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t\" (UID: \"ed107f59-f9a9-4a99-8a89-26a288611dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"
Oct 30 04:40:30.790922 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:30.790839    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"
Oct 30 04:40:31.112372 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:31.112340    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"]
Oct 30 04:40:31.112931 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:40:31.112908    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded107f59_f9a9_4a99_8a89_26a288611dd1.slice/crio-36ab5f0f1c97bf0ab97f14f9c8abb8c8be8e75f8c5d388a598c98cca47b1b2ba WatchSource:0}: Error finding container 36ab5f0f1c97bf0ab97f14f9c8abb8c8be8e75f8c5d388a598c98cca47b1b2ba: Status 404 returned error can't find the container with id 36ab5f0f1c97bf0ab97f14f9c8abb8c8be8e75f8c5d388a598c98cca47b1b2ba
Oct 30 04:40:31.355542 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:31.355496    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t" event={"ID":"ed107f59-f9a9-4a99-8a89-26a288611dd1","Type":"ContainerStarted","Data":"2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3"}
Oct 30 04:40:31.355732 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:31.355667    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"
Oct 30 04:40:31.355732 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:31.355694    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t" event={"ID":"ed107f59-f9a9-4a99-8a89-26a288611dd1","Type":"ContainerStarted","Data":"36ab5f0f1c97bf0ab97f14f9c8abb8c8be8e75f8c5d388a598c98cca47b1b2ba"}
Oct 30 04:40:31.401381 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:31.401265    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t" podStartSLOduration=1.401249003 podStartE2EDuration="1.401249003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:31.400142769 +0000 UTC m=+2611.141565867" watchObservedRunningTime="2026-10-30 04:40:31.401249003 +0000 UTC m=+2611.142672099"
Oct 30 04:40:37.217515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.217483    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"]
Oct 30 04:40:37.222733 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.222707    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.240680 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.240654    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"]
Oct 30 04:40:37.251987 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.251950    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-66217b\"/\"kube-root-ca.crt\""
Oct 30 04:40:37.251987 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.251969    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-66217b\"/\"default-dockercfg-pjgtt\""
Oct 30 04:40:37.252202 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.252004    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-66217b\"/\"auth-test-model-66217b-kserve-self-signed-certs\""
Oct 30 04:40:37.252202 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.252033    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-66217b\"/\"openshift-service-ca.crt\""
Oct 30 04:40:37.312227 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.312200    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-home\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.312383 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.312237    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tmp-dir\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.312383 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.312270    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpxk\" (UniqueName: \"kubernetes.io/projected/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-kube-api-access-mtpxk\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.312490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.312384    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tls-certs\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.312490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.312406    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-dshm\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.312490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.312436    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-model-cache\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.414057 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.414022    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-home\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.414288 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.414101    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tmp-dir\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.414288 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.414145    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpxk\" (UniqueName: \"kubernetes.io/projected/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-kube-api-access-mtpxk\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.414288 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.414195    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tls-certs\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.414288 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.414233    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-dshm\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.414288 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.414260    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-model-cache\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.414566 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.414526    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tmp-dir\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.414772 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.414744    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-home\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.414905 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.414757    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-model-cache\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.417035 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.417010    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tls-certs\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.417174 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.417116    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-dshm\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.438487 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.438461    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpxk\" (UniqueName: \"kubernetes.io/projected/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-kube-api-access-mtpxk\") pod \"auth-test-model-66217b-kserve-5c4f66b785-c96zz\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") " pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.539905 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.539805    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:37.845680 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:40:37.845652    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8fe6eaf_5aae_4ebe_9fe1_82ba7124039b.slice/crio-8ce257b4221131659f6dab37436fd8b2f7ee4cb1d302ad1591e0d79b6d6dc44f WatchSource:0}: Error finding container 8ce257b4221131659f6dab37436fd8b2f7ee4cb1d302ad1591e0d79b6d6dc44f: Status 404 returned error can't find the container with id 8ce257b4221131659f6dab37436fd8b2f7ee4cb1d302ad1591e0d79b6d6dc44f
Oct 30 04:40:37.845797 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:37.845731    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"]
Oct 30 04:40:38.409483 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:38.409443    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz" event={"ID":"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b","Type":"ContainerStarted","Data":"55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491"}
Oct 30 04:40:38.409483 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:38.409486    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz" event={"ID":"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b","Type":"ContainerStarted","Data":"8ce257b4221131659f6dab37436fd8b2f7ee4cb1d302ad1591e0d79b6d6dc44f"}
Oct 30 04:40:38.445162 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:38.445109    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz" podStartSLOduration=1.4450685810000001 podStartE2EDuration="1.445068581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:38.443638342 +0000 UTC m=+2618.185061440" watchObservedRunningTime="2026-10-30 04:40:38.445068581 +0000 UTC m=+2618.186491679"
Oct 30 04:40:38.540603 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:38.540576    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:39.622570 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:39.622537    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"]
Oct 30 04:40:39.623026 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:39.622753    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" podUID="53c7a3aa-110a-4656-9b44-c4a1cd611042" containerName="istio-proxy" containerID="cri-o://236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5" gracePeriod=30
Oct 30 04:40:41.752291 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:40:41.752255    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:40:42.364864 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:42.364836    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"
Oct 30 04:40:44.113166 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:44.113132    5072 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:40:44.113551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:44.113213    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" podUID="53c7a3aa-110a-4656-9b44-c4a1cd611042" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:40:44.886317 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:44.886287    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:40:45.002618 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.002589    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53c7a3aa-110a-4656-9b44-c4a1cd611042-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53c7a3aa-110a-4656-9b44-c4a1cd611042-istiod-ca-cert\") pod \"53c7a3aa-110a-4656-9b44-c4a1cd611042\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") "
Oct 30 04:40:45.002798 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.002665    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-kube-api-access-ch7f8\" (UniqueName: \"kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-kube-api-access-ch7f8\") pod \"53c7a3aa-110a-4656-9b44-c4a1cd611042\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") "
Oct 30 04:40:45.002798 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.002725    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-token\" (UniqueName: \"kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-token\") pod \"53c7a3aa-110a-4656-9b44-c4a1cd611042\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") "
Oct 30 04:40:45.002798 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.002764    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-envoy\") pod \"53c7a3aa-110a-4656-9b44-c4a1cd611042\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") "
Oct 30 04:40:45.002962 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.002798    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-podinfo\") pod \"53c7a3aa-110a-4656-9b44-c4a1cd611042\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") "
Oct 30 04:40:45.002962 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.002866    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-socket\") pod \"53c7a3aa-110a-4656-9b44-c4a1cd611042\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") "
Oct 30 04:40:45.002962 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.002905    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-credential-socket\") pod \"53c7a3aa-110a-4656-9b44-c4a1cd611042\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") "
Oct 30 04:40:45.002962 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.002936    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-data\") pod \"53c7a3aa-110a-4656-9b44-c4a1cd611042\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") "
Oct 30 04:40:45.003232 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.002971    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-certs\") pod \"53c7a3aa-110a-4656-9b44-c4a1cd611042\" (UID: \"53c7a3aa-110a-4656-9b44-c4a1cd611042\") "
Oct 30 04:40:45.003232 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.002993    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c7a3aa-110a-4656-9b44-c4a1cd611042-istiod-ca-cert" pod "53c7a3aa-110a-4656-9b44-c4a1cd611042" (UID: "53c7a3aa-110a-4656-9b44-c4a1cd611042"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:40:45.003349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.003238    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-socket" pod "53c7a3aa-110a-4656-9b44-c4a1cd611042" (UID: "53c7a3aa-110a-4656-9b44-c4a1cd611042"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:45.003349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.003338    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-credential-socket" pod "53c7a3aa-110a-4656-9b44-c4a1cd611042" (UID: "53c7a3aa-110a-4656-9b44-c4a1cd611042"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:45.003546 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.003513    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-data" pod "53c7a3aa-110a-4656-9b44-c4a1cd611042" (UID: "53c7a3aa-110a-4656-9b44-c4a1cd611042"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:45.003546 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.003542    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:40:45.003718 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.003561    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53c7a3aa-110a-4656-9b44-c4a1cd611042-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:40:45.003718 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.003574    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:40:45.003718 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.003706    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-certs" pod "53c7a3aa-110a-4656-9b44-c4a1cd611042" (UID: "53c7a3aa-110a-4656-9b44-c4a1cd611042"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:45.005139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.005117    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-kube-api-access-ch7f8" pod "53c7a3aa-110a-4656-9b44-c4a1cd611042" (UID: "53c7a3aa-110a-4656-9b44-c4a1cd611042"). InnerVolumeSpecName "kube-api-access-ch7f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:40:45.005245 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.005209    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-token" pod "53c7a3aa-110a-4656-9b44-c4a1cd611042" (UID: "53c7a3aa-110a-4656-9b44-c4a1cd611042"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:40:45.005638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.005615    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-podinfo" pod "53c7a3aa-110a-4656-9b44-c4a1cd611042" (UID: "53c7a3aa-110a-4656-9b44-c4a1cd611042"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:40:45.005734 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.005631    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-envoy" pod "53c7a3aa-110a-4656-9b44-c4a1cd611042" (UID: "53c7a3aa-110a-4656-9b44-c4a1cd611042"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:45.104973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.104941    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:40:45.104973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.104968    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:40:45.104973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.104977    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7f8\" (UniqueName: \"kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-kube-api-access-ch7f8\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:40:45.105291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.104987    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:40:45.105291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.104995    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:40:45.105291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.105002    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53c7a3aa-110a-4656-9b44-c4a1cd611042-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:40:45.450147 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.450118    5072 generic.go:356] "Generic (PLEG): container finished" podID="53c7a3aa-110a-4656-9b44-c4a1cd611042" containerID="236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5" exitCode=0
Oct 30 04:40:45.450645 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.450171    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"
Oct 30 04:40:45.450645 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.450206    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" event={"ID":"53c7a3aa-110a-4656-9b44-c4a1cd611042","Type":"ContainerDied","Data":"236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5"}
Oct 30 04:40:45.450645 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.450266    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29" event={"ID":"53c7a3aa-110a-4656-9b44-c4a1cd611042","Type":"ContainerDied","Data":"89a88444bb76afa1b02760fdcd6bdb85ac6949ac3000eff94dc56d3d58ae1d04"}
Oct 30 04:40:45.450645 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.450295    5072 scope.go:122] "RemoveContainer" containerID="236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5"
Oct 30 04:40:45.461635 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.461607    5072 scope.go:122] "RemoveContainer" containerID="236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5"
Oct 30 04:40:45.461870 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:40:45.461851    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5\": container with ID starting with 236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5 not found: ID does not exist" containerID="236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5"
Oct 30 04:40:45.461923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.461882    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5"} err="failed to get container status \"236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5\": rpc error: code = NotFound desc = could not find container \"236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5\": container with ID starting with 236405d5d199f3c332e2bc3e05b3828730f459f8884b6290cba54a5cb5a737e5 not found: ID does not exist"
Oct 30 04:40:45.519346 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.519317    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"]
Oct 30 04:40:45.531935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:45.531905    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-cnj29"]
Oct 30 04:40:46.407950 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:46.407923    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c7a3aa-110a-4656-9b44-c4a1cd611042" path="/var/lib/kubelet/pods/53c7a3aa-110a-4656-9b44-c4a1cd611042/volumes"
Oct 30 04:40:47.540544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:47.540509    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:47.553404 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:47.553380    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:48.479317 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:48.479286    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:40:56.733527 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.733491    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"]
Oct 30 04:40:56.734102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.734085    5072 cpu_manager.go:414] "containerMap: removing container" podUID="53c7a3aa-110a-4656-9b44-c4a1cd611042" containerName="istio-proxy"
Oct 30 04:40:56.734154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.734105    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c7a3aa-110a-4656-9b44-c4a1cd611042" containerName="istio-proxy"
Oct 30 04:40:56.734227 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.734218    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c7a3aa-110a-4656-9b44-c4a1cd611042" containerName="istio-proxy"
Oct 30 04:40:56.737592 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.737577    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"
Oct 30 04:40:56.755245 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.755216    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"]
Oct 30 04:40:56.765108 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.765044    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-4c2415\""
Oct 30 04:40:56.825971 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.825937    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"]
Oct 30 04:40:56.830980 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.830954    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-4c2415-dockercfg-fvv5b\""
Oct 30 04:40:56.831470 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.831442    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"
Oct 30 04:40:56.846656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.845521    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5027a9-d674-4f19-a204-7aeae7393585-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd\" (UID: \"fe5027a9-d674-4f19-a204-7aeae7393585\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"
Oct 30 04:40:56.847051 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.847019    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwkml\" (UniqueName: \"kubernetes.io/projected/fe5027a9-d674-4f19-a204-7aeae7393585-kube-api-access-dwkml\") pod \"payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd\" (UID: \"fe5027a9-d674-4f19-a204-7aeae7393585\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"
Oct 30 04:40:56.851126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.851100    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"]
Oct 30 04:40:56.947759 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.947723    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwkml\" (UniqueName: \"kubernetes.io/projected/fe5027a9-d674-4f19-a204-7aeae7393585-kube-api-access-dwkml\") pod \"payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd\" (UID: \"fe5027a9-d674-4f19-a204-7aeae7393585\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"
Oct 30 04:40:56.947759 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.947762    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t\" (UID: \"64ba6a9f-dabb-4827-bbda-7f89e33de8b8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"
Oct 30 04:40:56.948012 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.947837    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5027a9-d674-4f19-a204-7aeae7393585-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd\" (UID: \"fe5027a9-d674-4f19-a204-7aeae7393585\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"
Oct 30 04:40:56.948012 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.947859    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55dsp\" (UniqueName: \"kubernetes.io/projected/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-kube-api-access-55dsp\") pod \"payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t\" (UID: \"64ba6a9f-dabb-4827-bbda-7f89e33de8b8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"
Oct 30 04:40:56.948615 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.948591    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5027a9-d674-4f19-a204-7aeae7393585-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd\" (UID: \"fe5027a9-d674-4f19-a204-7aeae7393585\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"
Oct 30 04:40:56.971951 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:56.971921    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwkml\" (UniqueName: \"kubernetes.io/projected/fe5027a9-d674-4f19-a204-7aeae7393585-kube-api-access-dwkml\") pod \"payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd\" (UID: \"fe5027a9-d674-4f19-a204-7aeae7393585\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"
Oct 30 04:40:57.049313 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.049225    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55dsp\" (UniqueName: \"kubernetes.io/projected/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-kube-api-access-55dsp\") pod \"payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t\" (UID: \"64ba6a9f-dabb-4827-bbda-7f89e33de8b8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"
Oct 30 04:40:57.049476 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.049381    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t\" (UID: \"64ba6a9f-dabb-4827-bbda-7f89e33de8b8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"
Oct 30 04:40:57.050058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.050037    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t\" (UID: \"64ba6a9f-dabb-4827-bbda-7f89e33de8b8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"
Oct 30 04:40:57.052971 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.052951    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"
Oct 30 04:40:57.074367 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.074341    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55dsp\" (UniqueName: \"kubernetes.io/projected/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-kube-api-access-55dsp\") pod \"payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t\" (UID: \"64ba6a9f-dabb-4827-bbda-7f89e33de8b8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"
Oct 30 04:40:57.151322 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.150822    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"
Oct 30 04:40:57.170915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.170883    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"]
Oct 30 04:40:57.175655 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.175629    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.191815 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.190991    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"]
Oct 30 04:40:57.202922 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.202893    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e070ea\"/\"kube-root-ca.crt\""
Oct 30 04:40:57.203338 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.203313    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e070ea\"/\"auth-test-model-e070ea-kserve-self-signed-certs\""
Oct 30 04:40:57.203472 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.203421    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e070ea\"/\"openshift-service-ca.crt\""
Oct 30 04:40:57.203553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.203528    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e070ea\"/\"default-dockercfg-zx98d\""
Oct 30 04:40:57.251485 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.251449    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-model-cache\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.251635 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.251519    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-home\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.251635 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.251604    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-tmp-dir\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.251737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.251642    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl92n\" (UniqueName: \"kubernetes.io/projected/fe067687-49f8-4781-9c02-a2b47ea35293-kube-api-access-kl92n\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.251737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.251693    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe067687-49f8-4781-9c02-a2b47ea35293-tls-certs\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.251835 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.251751    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-dshm\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.353409 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.353326    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-home\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.353409 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.353399    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-tmp-dir\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.353724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.353423    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl92n\" (UniqueName: \"kubernetes.io/projected/fe067687-49f8-4781-9c02-a2b47ea35293-kube-api-access-kl92n\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.353724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.353453    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe067687-49f8-4781-9c02-a2b47ea35293-tls-certs\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.353724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.353486    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-dshm\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.353724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.353563    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-model-cache\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.353975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.353806    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-home\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.353975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.353844    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-tmp-dir\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.353975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.353940    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-model-cache\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.356605 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.356501    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-dshm\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.357157 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.356913    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe067687-49f8-4781-9c02-a2b47ea35293-tls-certs\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.358731 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.358706    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"]
Oct 30 04:40:57.359641 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:40:57.359609    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5027a9_d674_4f19_a204_7aeae7393585.slice/crio-d6f5455804fcdd374ce6a02e5b352f0a78b429de9a64c8df09a5190cfecbf015 WatchSource:0}: Error finding container d6f5455804fcdd374ce6a02e5b352f0a78b429de9a64c8df09a5190cfecbf015: Status 404 returned error can't find the container with id d6f5455804fcdd374ce6a02e5b352f0a78b429de9a64c8df09a5190cfecbf015
Oct 30 04:40:57.378310 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.378286    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl92n\" (UniqueName: \"kubernetes.io/projected/fe067687-49f8-4781-9c02-a2b47ea35293-kube-api-access-kl92n\") pod \"auth-test-model-e070ea-kserve-7b84868f5c-9wx26\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.477803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.477768    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"]
Oct 30 04:40:57.479573 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:40:57.479537    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ba6a9f_dabb_4827_bbda_7f89e33de8b8.slice/crio-661b42ce8793716f1fe800a9da7e2a4f07df9076bd04cbd0b38838b3a21d436e WatchSource:0}: Error finding container 661b42ce8793716f1fe800a9da7e2a4f07df9076bd04cbd0b38838b3a21d436e: Status 404 returned error can't find the container with id 661b42ce8793716f1fe800a9da7e2a4f07df9076bd04cbd0b38838b3a21d436e
Oct 30 04:40:57.514177 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.514145    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:40:57.534043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.534014    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t" event={"ID":"64ba6a9f-dabb-4827-bbda-7f89e33de8b8","Type":"ContainerStarted","Data":"661b42ce8793716f1fe800a9da7e2a4f07df9076bd04cbd0b38838b3a21d436e"}
Oct 30 04:40:57.536174 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.536147    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd" event={"ID":"fe5027a9-d674-4f19-a204-7aeae7393585","Type":"ContainerStarted","Data":"b1e9a62dc86601894e9669577c8853a9a950a6d2cc9bc9ee5be71b88fab48247"}
Oct 30 04:40:57.536269 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.536201    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd" event={"ID":"fe5027a9-d674-4f19-a204-7aeae7393585","Type":"ContainerStarted","Data":"d6f5455804fcdd374ce6a02e5b352f0a78b429de9a64c8df09a5190cfecbf015"}
Oct 30 04:40:57.536330 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.536313    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"
Oct 30 04:40:57.583021 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.582965    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd" podStartSLOduration=1.5829464199999999 podStartE2EDuration="1.58294642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:57.578996663 +0000 UTC m=+2637.320419761" watchObservedRunningTime="2026-10-30 04:40:57.58294642 +0000 UTC m=+2637.324369518"
Oct 30 04:40:57.824921 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:40:57.824887    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe067687_49f8_4781_9c02_a2b47ea35293.slice/crio-a8bd9a9c61c1fce8da136fd8ccbeb570d109d658d5d49d7554d7ea0fa7c15117 WatchSource:0}: Error finding container a8bd9a9c61c1fce8da136fd8ccbeb570d109d658d5d49d7554d7ea0fa7c15117: Status 404 returned error can't find the container with id a8bd9a9c61c1fce8da136fd8ccbeb570d109d658d5d49d7554d7ea0fa7c15117
Oct 30 04:40:57.825319 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:57.825285    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"]
Oct 30 04:40:58.543901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:58.543861    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26" event={"ID":"fe067687-49f8-4781-9c02-a2b47ea35293","Type":"ContainerStarted","Data":"3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334"}
Oct 30 04:40:58.544116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:58.543916    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26" event={"ID":"fe067687-49f8-4781-9c02-a2b47ea35293","Type":"ContainerStarted","Data":"a8bd9a9c61c1fce8da136fd8ccbeb570d109d658d5d49d7554d7ea0fa7c15117"}
Oct 30 04:40:58.545804 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:58.545777    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t" event={"ID":"64ba6a9f-dabb-4827-bbda-7f89e33de8b8","Type":"ContainerStarted","Data":"28dfc0622c7bbeb42a9a2d5823b1548450912f812a6c339484b6707e3a23868a"}
Oct 30 04:40:58.545924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:58.545846    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"
Oct 30 04:40:58.589276 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:58.589229    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26" podStartSLOduration=1.5892142470000001 podStartE2EDuration="1.589214247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:58.588889192 +0000 UTC m=+2638.330312290" watchObservedRunningTime="2026-10-30 04:40:58.589214247 +0000 UTC m=+2638.330637345"
Oct 30 04:40:58.633900 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:58.633839    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t" podStartSLOduration=2.633826068 podStartE2EDuration="2.633826068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:58.630297537 +0000 UTC m=+2638.371720636" watchObservedRunningTime="2026-10-30 04:40:58.633826068 +0000 UTC m=+2638.375249161"
Oct 30 04:40:59.514739 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:40:59.514708    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:41:07.514588 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:07.514549    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:41:07.526869 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:07.526846    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:41:07.613813 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:07.613783    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:41:08.548802 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:08.548770    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"
Oct 30 04:41:09.555853 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:09.555827    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"
Oct 30 04:41:11.752604 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:41:11.752562    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:41:16.319301 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.319259    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"]
Oct 30 04:41:16.325489 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.325465    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.342286 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.342257    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"]
Oct 30 04:41:16.351829 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.351805    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-e6b837\"/\"openshift-service-ca.crt\""
Oct 30 04:41:16.351829 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.351802    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-e6b837\"/\"kube-root-ca.crt\""
Oct 30 04:41:16.352069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.352055    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-e6b837\"/\"auth-test-model-e6b837-kserve-self-signed-certs\""
Oct 30 04:41:16.355063 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.355039    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-e6b837\"/\"default-dockercfg-xskpd\""
Oct 30 04:41:16.472968 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.472928    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tls-certs\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.473228 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.473089    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts6vr\" (UniqueName: \"kubernetes.io/projected/d34726f4-eae7-4909-b6d0-a3b341f4f13d-kube-api-access-ts6vr\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.473228 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.473138    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-model-cache\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.473228 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.473197    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tmp-dir\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.473425 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.473242    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-home\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.473425 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.473418    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-dshm\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.574473 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.574380    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-dshm\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.574638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.574503    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tls-certs\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.574638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.574573    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts6vr\" (UniqueName: \"kubernetes.io/projected/d34726f4-eae7-4909-b6d0-a3b341f4f13d-kube-api-access-ts6vr\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.574638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.574609    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-model-cache\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.574638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.574634    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tmp-dir\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.574870 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.574680    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-home\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.575029 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.574996    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tmp-dir\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.575258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.575071    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-home\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.575511 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.575486    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-model-cache\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.576849 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.576828    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-dshm\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.577370 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.577348    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tls-certs\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.599809 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.599748    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts6vr\" (UniqueName: \"kubernetes.io/projected/d34726f4-eae7-4909-b6d0-a3b341f4f13d-kube-api-access-ts6vr\") pod \"auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.639977 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.639950    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:16.949556 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:41:16.949525    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd34726f4_eae7_4909_b6d0_a3b341f4f13d.slice/crio-55892da10f655e52db85bbcf72a8ad26f3b125220797d560b2ab585731e8aef2 WatchSource:0}: Error finding container 55892da10f655e52db85bbcf72a8ad26f3b125220797d560b2ab585731e8aef2: Status 404 returned error can't find the container with id 55892da10f655e52db85bbcf72a8ad26f3b125220797d560b2ab585731e8aef2
Oct 30 04:41:16.950953 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:16.950922    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"]
Oct 30 04:41:17.685034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:17.684995    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h" event={"ID":"d34726f4-eae7-4909-b6d0-a3b341f4f13d","Type":"ContainerStarted","Data":"b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd"}
Oct 30 04:41:17.685034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:17.685039    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h" event={"ID":"d34726f4-eae7-4909-b6d0-a3b341f4f13d","Type":"ContainerStarted","Data":"55892da10f655e52db85bbcf72a8ad26f3b125220797d560b2ab585731e8aef2"}
Oct 30 04:41:17.728932 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:17.728883    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h" podStartSLOduration=1.728863601 podStartE2EDuration="1.728863601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:41:17.727338699 +0000 UTC m=+2657.468761798" watchObservedRunningTime="2026-10-30 04:41:17.728863601 +0000 UTC m=+2657.470286698"
Oct 30 04:41:18.641089 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:18.641057    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:25.673541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:25.673499    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"]
Oct 30 04:41:25.677865 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:25.677847    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"
Oct 30 04:41:25.695109 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:25.695066    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"]
Oct 30 04:41:25.705070 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:25.705048    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-20a650\""
Oct 30 04:41:25.771557 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:25.771522    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cc95\" (UniqueName: \"kubernetes.io/projected/5bb9b08f-8c29-48ea-abe1-8ab303a46756-kube-api-access-9cc95\") pod \"payload-processing-e2e-explicit-20a650-7f7568947d-v6f87\" (UID: \"5bb9b08f-8c29-48ea-abe1-8ab303a46756\") " pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"
Oct 30 04:41:25.771795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:25.771768    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb9b08f-8c29-48ea-abe1-8ab303a46756-plugins-config-volume\") pod \"payload-processing-e2e-explicit-20a650-7f7568947d-v6f87\" (UID: \"5bb9b08f-8c29-48ea-abe1-8ab303a46756\") " pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"
Oct 30 04:41:25.872940 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:25.872903    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cc95\" (UniqueName: \"kubernetes.io/projected/5bb9b08f-8c29-48ea-abe1-8ab303a46756-kube-api-access-9cc95\") pod \"payload-processing-e2e-explicit-20a650-7f7568947d-v6f87\" (UID: \"5bb9b08f-8c29-48ea-abe1-8ab303a46756\") " pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"
Oct 30 04:41:25.873140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:25.873114    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb9b08f-8c29-48ea-abe1-8ab303a46756-plugins-config-volume\") pod \"payload-processing-e2e-explicit-20a650-7f7568947d-v6f87\" (UID: \"5bb9b08f-8c29-48ea-abe1-8ab303a46756\") " pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"
Oct 30 04:41:25.874022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:25.873992    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb9b08f-8c29-48ea-abe1-8ab303a46756-plugins-config-volume\") pod \"payload-processing-e2e-explicit-20a650-7f7568947d-v6f87\" (UID: \"5bb9b08f-8c29-48ea-abe1-8ab303a46756\") " pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"
Oct 30 04:41:25.896744 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:25.896721    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cc95\" (UniqueName: \"kubernetes.io/projected/5bb9b08f-8c29-48ea-abe1-8ab303a46756-kube-api-access-9cc95\") pod \"payload-processing-e2e-explicit-20a650-7f7568947d-v6f87\" (UID: \"5bb9b08f-8c29-48ea-abe1-8ab303a46756\") " pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"
Oct 30 04:41:25.993248 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:25.993220    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"
Oct 30 04:41:26.295433 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:26.295402    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"]
Oct 30 04:41:26.296071 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:41:26.296043    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb9b08f_8c29_48ea_abe1_8ab303a46756.slice/crio-c0a6a3fe9492fdec1735154217447479a28ff9e3733b9104cf648284404819fb WatchSource:0}: Error finding container c0a6a3fe9492fdec1735154217447479a28ff9e3733b9104cf648284404819fb: Status 404 returned error can't find the container with id c0a6a3fe9492fdec1735154217447479a28ff9e3733b9104cf648284404819fb
Oct 30 04:41:26.640636 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:26.640556    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:26.652922 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:26.652900    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:26.739139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:26.739104    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87" event={"ID":"5bb9b08f-8c29-48ea-abe1-8ab303a46756","Type":"ContainerStarted","Data":"be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4"}
Oct 30 04:41:26.739599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:26.739158    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87" event={"ID":"5bb9b08f-8c29-48ea-abe1-8ab303a46756","Type":"ContainerStarted","Data":"c0a6a3fe9492fdec1735154217447479a28ff9e3733b9104cf648284404819fb"}
Oct 30 04:41:26.739599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:26.739243    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"
Oct 30 04:41:26.750882 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:26.750858    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:41:26.782841 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:26.782793    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87" podStartSLOduration=1.7827809559999999 podStartE2EDuration="1.782780956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:41:26.780183483 +0000 UTC m=+2666.521606582" watchObservedRunningTime="2026-10-30 04:41:26.782780956 +0000 UTC m=+2666.524204088"
Oct 30 04:41:37.749728 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:37.749701    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"
Oct 30 04:41:39.518565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.518527    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"]
Oct 30 04:41:39.524447 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.524415    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.541998 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.541970    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"]
Oct 30 04:41:39.550853 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.550806    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-247158\"/\"kube-root-ca.crt\""
Oct 30 04:41:39.551050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.550839    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-247158\"/\"openshift-service-ca.crt\""
Oct 30 04:41:39.558543 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.558520    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-247158\"/\"default-dockercfg-g7h4m\""
Oct 30 04:41:39.558646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.558544    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-247158\"/\"auth-test-model-247158-kserve-self-signed-certs\""
Oct 30 04:41:39.628489 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.628457    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-home\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.628489 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.628489    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-dshm\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.628731 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.628572    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44lcz\" (UniqueName: \"kubernetes.io/projected/8b1ad519-9858-4903-847c-e8dc14645411-kube-api-access-44lcz\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.628731 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.628613    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1ad519-9858-4903-847c-e8dc14645411-tls-certs\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.628731 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.628692    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-tmp-dir\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.628891 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.628758    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-model-cache\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.730193 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.730157    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-home\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.730396 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.730202    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-dshm\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.730396 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.730288    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lcz\" (UniqueName: \"kubernetes.io/projected/8b1ad519-9858-4903-847c-e8dc14645411-kube-api-access-44lcz\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.730396 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.730316    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1ad519-9858-4903-847c-e8dc14645411-tls-certs\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.730396 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.730389    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-tmp-dir\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.730635 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.730483    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-model-cache\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.730635 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.730616    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-home\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.730884 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.730863    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-model-cache\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.730968 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.730945    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-tmp-dir\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.733418 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.733394    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-dshm\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.733690 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.733669    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1ad519-9858-4903-847c-e8dc14645411-tls-certs\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.754968 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.754937    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44lcz\" (UniqueName: \"kubernetes.io/projected/8b1ad519-9858-4903-847c-e8dc14645411-kube-api-access-44lcz\") pod \"auth-test-model-247158-kserve-78bc8ff669-hq2dk\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") " pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:39.838822 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:39.838735    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:40.152461 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:40.152437    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"]
Oct 30 04:41:40.152846 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:41:40.152823    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b1ad519_9858_4903_847c_e8dc14645411.slice/crio-848519b6e7606b62b2b14c51e0b96c36ceb21a2179c79f00882cb95b9dbb50ee WatchSource:0}: Error finding container 848519b6e7606b62b2b14c51e0b96c36ceb21a2179c79f00882cb95b9dbb50ee: Status 404 returned error can't find the container with id 848519b6e7606b62b2b14c51e0b96c36ceb21a2179c79f00882cb95b9dbb50ee
Oct 30 04:41:40.832322 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:40.832286    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk" event={"ID":"8b1ad519-9858-4903-847c-e8dc14645411","Type":"ContainerStarted","Data":"ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b"}
Oct 30 04:41:40.832727 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:40.832330    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk" event={"ID":"8b1ad519-9858-4903-847c-e8dc14645411","Type":"ContainerStarted","Data":"848519b6e7606b62b2b14c51e0b96c36ceb21a2179c79f00882cb95b9dbb50ee"}
Oct 30 04:41:40.839580 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:40.839555    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:40.876256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:40.876212    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk" podStartSLOduration=1.8762000730000001 podStartE2EDuration="1.876200073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:41:40.874906377 +0000 UTC m=+2680.616329475" watchObservedRunningTime="2026-10-30 04:41:40.876200073 +0000 UTC m=+2680.617623170"
Oct 30 04:41:41.757659 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:41:41.757622    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:41:49.839571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:49.839527    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:49.853684 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:49.853659    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:41:49.900533 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:41:49.900502    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:42:00.756022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:00.755901    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:42:00.773005 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:00.772984    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:42:11.752196 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:42:11.752146    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:42:17.032961 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.032920    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"]
Oct 30 04:42:17.037635 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.037610    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.062169 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.062145    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"]
Oct 30 04:42:17.065712 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.065681    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/632b903f-f18c-4d75-b439-c0006dfe5d27-tls-certs\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.065867 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.065774    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-home\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.065867 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.065799    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-tmp-dir\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.065867 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.065852    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-model-cache\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.066133 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.065885    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mwvp\" (UniqueName: \"kubernetes.io/projected/632b903f-f18c-4d75-b439-c0006dfe5d27-kube-api-access-2mwvp\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.066200 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.066178    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-dshm\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.071671 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.071650    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-20a650\"/\"openshift-service-ca.crt\""
Oct 30 04:42:17.071807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.071731    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-20a650\"/\"kube-root-ca.crt\""
Oct 30 04:42:17.071907 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.071827    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-20a650\"/\"default-dockercfg-98xgl\""
Oct 30 04:42:17.071907 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.071878    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-20a650\"/\"explicit-model-20a650-kserve-self-signed-certs\""
Oct 30 04:42:17.167068 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.167032    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/632b903f-f18c-4d75-b439-c0006dfe5d27-tls-certs\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.167365 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.167150    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-home\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.167365 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.167172    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-tmp-dir\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.167365 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.167212    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-model-cache\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.167365 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.167242    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mwvp\" (UniqueName: \"kubernetes.io/projected/632b903f-f18c-4d75-b439-c0006dfe5d27-kube-api-access-2mwvp\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.167365 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.167310    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-dshm\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.167912 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.167890    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-home\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.168012 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.167948    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-model-cache\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.168199 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.168179    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-tmp-dir\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.169952 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.169934    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-dshm\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.170046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.170025    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/632b903f-f18c-4d75-b439-c0006dfe5d27-tls-certs\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.195062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.195034    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mwvp\" (UniqueName: \"kubernetes.io/projected/632b903f-f18c-4d75-b439-c0006dfe5d27-kube-api-access-2mwvp\") pod \"explicit-model-20a650-kserve-749b8cb84c-4rft2\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") " pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.352392 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.352307    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:17.660417 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:17.660385    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"]
Oct 30 04:42:17.660843 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:42:17.660819    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod632b903f_f18c_4d75_b439_c0006dfe5d27.slice/crio-be1714a7f5472f97fa7815c72cf22c5171fb7d0664d82ccfce4130371fa9753b WatchSource:0}: Error finding container be1714a7f5472f97fa7815c72cf22c5171fb7d0664d82ccfce4130371fa9753b: Status 404 returned error can't find the container with id be1714a7f5472f97fa7815c72cf22c5171fb7d0664d82ccfce4130371fa9753b
Oct 30 04:42:18.077875 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:18.077837    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2" event={"ID":"632b903f-f18c-4d75-b439-c0006dfe5d27","Type":"ContainerStarted","Data":"867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156"}
Oct 30 04:42:18.078280 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:18.077886    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2" event={"ID":"632b903f-f18c-4d75-b439-c0006dfe5d27","Type":"ContainerStarted","Data":"be1714a7f5472f97fa7815c72cf22c5171fb7d0664d82ccfce4130371fa9753b"}
Oct 30 04:42:18.123770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:18.123709    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2" podStartSLOduration=1.123690338 podStartE2EDuration="1.123690338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:42:18.121179591 +0000 UTC m=+2717.862602685" watchObservedRunningTime="2026-10-30 04:42:18.123690338 +0000 UTC m=+2717.865113436"
Oct 30 04:42:18.352949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:18.352880    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:27.353397 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:27.353366    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:27.366225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:27.366195    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:28.151873 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:28.151841    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:30.628672 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:30.628616    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"]
Oct 30 04:42:41.754837 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:42:41.754748    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:42:45.657019 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:45.656959    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2" podUID="632b903f-f18c-4d75-b439-c0006dfe5d27" containerName="main" containerID="cri-o://867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156" gracePeriod=45
Oct 30 04:42:45.937469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:45.937445    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:46.090488 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.090457    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632b903f-f18c-4d75-b439-c0006dfe5d27-kube-api-access-2mwvp\" (UniqueName: \"kubernetes.io/projected/632b903f-f18c-4d75-b439-c0006dfe5d27-kube-api-access-2mwvp\") pod \"632b903f-f18c-4d75-b439-c0006dfe5d27\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") "
Oct 30 04:42:46.090650 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.090508    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-home\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-home\") pod \"632b903f-f18c-4d75-b439-c0006dfe5d27\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") "
Oct 30 04:42:46.090650 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.090540    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/632b903f-f18c-4d75-b439-c0006dfe5d27-tls-certs\" (UniqueName: \"kubernetes.io/secret/632b903f-f18c-4d75-b439-c0006dfe5d27-tls-certs\") pod \"632b903f-f18c-4d75-b439-c0006dfe5d27\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") "
Oct 30 04:42:46.090650 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.090632    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-tmp-dir\") pod \"632b903f-f18c-4d75-b439-c0006dfe5d27\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") "
Oct 30 04:42:46.090833 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.090814    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-model-cache\") pod \"632b903f-f18c-4d75-b439-c0006dfe5d27\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") "
Oct 30 04:42:46.090893 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.090816    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-home" pod "632b903f-f18c-4d75-b439-c0006dfe5d27" (UID: "632b903f-f18c-4d75-b439-c0006dfe5d27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:42:46.090962 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.090874    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-dshm\") pod \"632b903f-f18c-4d75-b439-c0006dfe5d27\" (UID: \"632b903f-f18c-4d75-b439-c0006dfe5d27\") "
Oct 30 04:42:46.091019 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.090940    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-tmp-dir" pod "632b903f-f18c-4d75-b439-c0006dfe5d27" (UID: "632b903f-f18c-4d75-b439-c0006dfe5d27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:42:46.091019 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.090998    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-model-cache" pod "632b903f-f18c-4d75-b439-c0006dfe5d27" (UID: "632b903f-f18c-4d75-b439-c0006dfe5d27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:42:46.091399 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.091375    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:42:46.091399 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.091401    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:42:46.091569 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.091413    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:42:46.093033 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.093009    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-dshm" pod "632b903f-f18c-4d75-b439-c0006dfe5d27" (UID: "632b903f-f18c-4d75-b439-c0006dfe5d27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:42:46.093180 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.093092    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632b903f-f18c-4d75-b439-c0006dfe5d27-kube-api-access-2mwvp" pod "632b903f-f18c-4d75-b439-c0006dfe5d27" (UID: "632b903f-f18c-4d75-b439-c0006dfe5d27"). InnerVolumeSpecName "kube-api-access-2mwvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:42:46.093180 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.093134    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/632b903f-f18c-4d75-b439-c0006dfe5d27-tls-certs" pod "632b903f-f18c-4d75-b439-c0006dfe5d27" (UID: "632b903f-f18c-4d75-b439-c0006dfe5d27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:42:46.193009 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.192982    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/632b903f-f18c-4d75-b439-c0006dfe5d27-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:42:46.193009 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.193007    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mwvp\" (UniqueName: \"kubernetes.io/projected/632b903f-f18c-4d75-b439-c0006dfe5d27-kube-api-access-2mwvp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:42:46.193204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.193017    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/632b903f-f18c-4d75-b439-c0006dfe5d27-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:42:46.255037 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.255009    5072 generic.go:356] "Generic (PLEG): container finished" podID="632b903f-f18c-4d75-b439-c0006dfe5d27" containerID="867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156" exitCode=0
Oct 30 04:42:46.255212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.255054    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"
Oct 30 04:42:46.255212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.255105    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2" event={"ID":"632b903f-f18c-4d75-b439-c0006dfe5d27","Type":"ContainerDied","Data":"867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156"}
Oct 30 04:42:46.255212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.255151    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2" event={"ID":"632b903f-f18c-4d75-b439-c0006dfe5d27","Type":"ContainerDied","Data":"be1714a7f5472f97fa7815c72cf22c5171fb7d0664d82ccfce4130371fa9753b"}
Oct 30 04:42:46.255212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.255169    5072 scope.go:122] "RemoveContainer" containerID="867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156"
Oct 30 04:42:46.267878 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.267853    5072 scope.go:122] "RemoveContainer" containerID="867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156"
Oct 30 04:42:46.268179 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:42:46.268160    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156\": container with ID starting with 867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156 not found: ID does not exist" containerID="867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156"
Oct 30 04:42:46.268226 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.268188    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156"} err="failed to get container status \"867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156\": rpc error: code = NotFound desc = could not find container \"867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156\": container with ID starting with 867397e22bd29b7c7bc1955d9c3e00c9fb887f02824b636bb2dae2bee3bab156 not found: ID does not exist"
Oct 30 04:42:46.304537 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.304512    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"]
Oct 30 04:42:46.315823 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.315798    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-20a650/explicit-model-20a650-kserve-749b8cb84c-4rft2"]
Oct 30 04:42:46.409475 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:46.409446    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="632b903f-f18c-4d75-b439-c0006dfe5d27" path="/var/lib/kubelet/pods/632b903f-f18c-4d75-b439-c0006dfe5d27/volumes"
Oct 30 04:42:50.776243 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.776206    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"]
Oct 30 04:42:50.776851 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.776836    5072 cpu_manager.go:414] "containerMap: removing container" podUID="632b903f-f18c-4d75-b439-c0006dfe5d27" containerName="main"
Oct 30 04:42:50.776900 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.776852    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632b903f-f18c-4d75-b439-c0006dfe5d27" containerName="main"
Oct 30 04:42:50.776983 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.776974    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="632b903f-f18c-4d75-b439-c0006dfe5d27" containerName="main"
Oct 30 04:42:50.781790 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.781767    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.801789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.801764    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"]
Oct 30 04:42:50.810646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.810612    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-4c2415\"/\"openshift-service-ca.crt\""
Oct 30 04:42:50.810646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.810627    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-4c2415\"/\"kube-root-ca.crt\""
Oct 30 04:42:50.811559 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.811540    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-4c2415\"/\"default-dockercfg-8gt4x\""
Oct 30 04:42:50.811696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.811601    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-4c2415\"/\"auth-test-model-4c2415-kserve-self-signed-certs\""
Oct 30 04:42:50.849284 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.849245    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-model-cache\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.849413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.849303    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9shn\" (UniqueName: \"kubernetes.io/projected/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-kube-api-access-r9shn\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.849413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.849372    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-home\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.849490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.849417    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tls-certs\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.849576 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.849556    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tmp-dir\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.849637 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.849621    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-dshm\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.951213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.951177    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tmp-dir\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.951423 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.951283    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-dshm\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.951423 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.951331    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-model-cache\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.951423 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.951376    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9shn\" (UniqueName: \"kubernetes.io/projected/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-kube-api-access-r9shn\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.951423 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.951420    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-home\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.951657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.951465    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tls-certs\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.951657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.951631    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tmp-dir\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.952399 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.952268    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-home\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.952399 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.952281    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-model-cache\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.953965 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.953933    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-dshm\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.954182 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.954165    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tls-certs\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:50.975584 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:50.975556    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9shn\" (UniqueName: \"kubernetes.io/projected/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-kube-api-access-r9shn\") pod \"auth-test-model-4c2415-kserve-748fb5cf57-jr4q5\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:51.096308 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:51.096214    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:42:51.405658 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:51.405630    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"]
Oct 30 04:42:51.405812 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:42:51.405788    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b01d3a7_44b9_4b1c_b6a6_f3ed540e2c9c.slice/crio-0eefe5a972b511013639768b8dc5f3e0118c3307ee22c61cc701746a90aff5e7 WatchSource:0}: Error finding container 0eefe5a972b511013639768b8dc5f3e0118c3307ee22c61cc701746a90aff5e7: Status 404 returned error can't find the container with id 0eefe5a972b511013639768b8dc5f3e0118c3307ee22c61cc701746a90aff5e7
Oct 30 04:42:52.294604 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:52.294571    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5" event={"ID":"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c","Type":"ContainerStarted","Data":"fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880"}
Oct 30 04:42:52.294991 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:52.294614    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5" event={"ID":"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c","Type":"ContainerStarted","Data":"0eefe5a972b511013639768b8dc5f3e0118c3307ee22c61cc701746a90aff5e7"}
Oct 30 04:42:52.339491 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:52.339434    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5" podStartSLOduration=2.339420551 podStartE2EDuration="2.339420551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:42:52.336508893 +0000 UTC m=+2752.077931993" watchObservedRunningTime="2026-10-30 04:42:52.339420551 +0000 UTC m=+2752.080843648"
Oct 30 04:42:53.097199 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:42:53.097168    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:43:01.096847 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:01.096810    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:43:01.110641 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:01.110615    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:43:01.365140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:01.365046    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:43:06.256451 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.256412    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"]
Oct 30 04:43:06.261820 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.261799    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:06.279146 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.279122    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"]
Oct 30 04:43:06.288376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.288335    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:43:06.288524 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.288406    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:43:06.431316 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.431288    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692nd\" (UniqueName: \"kubernetes.io/projected/002b0525-c6d4-4305-bb59-e3e470aaaff3-kube-api-access-692nd\") pod \"maas-api-revoke-sub-d58b6b8b-srgh5\" (UID: \"002b0525-c6d4-4305-bb59-e3e470aaaff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:06.431521 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.431501    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d58b6b8b-srgh5\" (UID: \"002b0525-c6d4-4305-bb59-e3e470aaaff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:06.431592 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.431538    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d58b6b8b-srgh5\" (UID: \"002b0525-c6d4-4305-bb59-e3e470aaaff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:06.532702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.532627    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d58b6b8b-srgh5\" (UID: \"002b0525-c6d4-4305-bb59-e3e470aaaff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:06.532702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.532663    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d58b6b8b-srgh5\" (UID: \"002b0525-c6d4-4305-bb59-e3e470aaaff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:06.532881 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.532706    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692nd\" (UniqueName: \"kubernetes.io/projected/002b0525-c6d4-4305-bb59-e3e470aaaff3-kube-api-access-692nd\") pod \"maas-api-revoke-sub-d58b6b8b-srgh5\" (UID: \"002b0525-c6d4-4305-bb59-e3e470aaaff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:06.533401 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.533372    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d58b6b8b-srgh5\" (UID: \"002b0525-c6d4-4305-bb59-e3e470aaaff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:06.535193 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.535174    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d58b6b8b-srgh5\" (UID: \"002b0525-c6d4-4305-bb59-e3e470aaaff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:06.563626 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.563604    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-692nd\" (UniqueName: \"kubernetes.io/projected/002b0525-c6d4-4305-bb59-e3e470aaaff3-kube-api-access-692nd\") pod \"maas-api-revoke-sub-d58b6b8b-srgh5\" (UID: \"002b0525-c6d4-4305-bb59-e3e470aaaff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:06.575492 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.575468    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:06.876222 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:43:06.876194    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002b0525_c6d4_4305_bb59_e3e470aaaff3.slice/crio-85167a3030e4ce8cab91c56ea60be85bbbd8701d7981ea473d451611a70d384e WatchSource:0}: Error finding container 85167a3030e4ce8cab91c56ea60be85bbbd8701d7981ea473d451611a70d384e: Status 404 returned error can't find the container with id 85167a3030e4ce8cab91c56ea60be85bbbd8701d7981ea473d451611a70d384e
Oct 30 04:43:06.876602 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:06.876576    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"]
Oct 30 04:43:07.398518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:07.398481    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5" event={"ID":"002b0525-c6d4-4305-bb59-e3e470aaaff3","Type":"ContainerStarted","Data":"43debf8a10f4b97ca035dc106637f204dab33f8131b1fcfff9378e5a282b4ea9"}
Oct 30 04:43:07.398937 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:07.398544    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5" event={"ID":"002b0525-c6d4-4305-bb59-e3e470aaaff3","Type":"ContainerStarted","Data":"85167a3030e4ce8cab91c56ea60be85bbbd8701d7981ea473d451611a70d384e"}
Oct 30 04:43:07.441961 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:07.441919    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5" podStartSLOduration=1.44190796 podStartE2EDuration="1.44190796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:07.439966289 +0000 UTC m=+2767.181389390" watchObservedRunningTime="2026-10-30 04:43:07.44190796 +0000 UTC m=+2767.183331078"
Oct 30 04:43:08.406522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:08.406483    5072 generic.go:356] "Generic (PLEG): container finished" podID="002b0525-c6d4-4305-bb59-e3e470aaaff3" containerID="43debf8a10f4b97ca035dc106637f204dab33f8131b1fcfff9378e5a282b4ea9" exitCode=0
Oct 30 04:43:08.408927 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:08.408895    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5" event={"ID":"002b0525-c6d4-4305-bb59-e3e470aaaff3","Type":"ContainerDied","Data":"43debf8a10f4b97ca035dc106637f204dab33f8131b1fcfff9378e5a282b4ea9"}
Oct 30 04:43:09.579705 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:09.579671    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:09.670207 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:09.670132    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/002b0525-c6d4-4305-bb59-e3e470aaaff3-kube-api-access-692nd\" (UniqueName: \"kubernetes.io/projected/002b0525-c6d4-4305-bb59-e3e470aaaff3-kube-api-access-692nd\") pod \"002b0525-c6d4-4305-bb59-e3e470aaaff3\" (UID: \"002b0525-c6d4-4305-bb59-e3e470aaaff3\") "
Oct 30 04:43:09.670207 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:09.670187    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-openshift-ca\") pod \"002b0525-c6d4-4305-bb59-e3e470aaaff3\" (UID: \"002b0525-c6d4-4305-bb59-e3e470aaaff3\") "
Oct 30 04:43:09.670430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:09.670245    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-xks-ca\") pod \"002b0525-c6d4-4305-bb59-e3e470aaaff3\" (UID: \"002b0525-c6d4-4305-bb59-e3e470aaaff3\") "
Oct 30 04:43:09.670638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:09.670609    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-openshift-ca" pod "002b0525-c6d4-4305-bb59-e3e470aaaff3" (UID: "002b0525-c6d4-4305-bb59-e3e470aaaff3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:43:09.670721 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:09.670682    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:09.672513 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:09.672488    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002b0525-c6d4-4305-bb59-e3e470aaaff3-kube-api-access-692nd" pod "002b0525-c6d4-4305-bb59-e3e470aaaff3" (UID: "002b0525-c6d4-4305-bb59-e3e470aaaff3"). InnerVolumeSpecName "kube-api-access-692nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:09.672513 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:09.672492    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-xks-ca" pod "002b0525-c6d4-4305-bb59-e3e470aaaff3" (UID: "002b0525-c6d4-4305-bb59-e3e470aaaff3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:43:09.772334 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:09.772305    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002b0525-c6d4-4305-bb59-e3e470aaaff3-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:09.772334 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:09.772331    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-692nd\" (UniqueName: \"kubernetes.io/projected/002b0525-c6d4-4305-bb59-e3e470aaaff3-kube-api-access-692nd\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:10.421889 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:10.421866    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"
Oct 30 04:43:10.422017 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:10.421913    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5" event={"ID":"002b0525-c6d4-4305-bb59-e3e470aaaff3","Type":"ContainerDied","Data":"85167a3030e4ce8cab91c56ea60be85bbbd8701d7981ea473d451611a70d384e"}
Oct 30 04:43:10.422017 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:10.421968    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85167a3030e4ce8cab91c56ea60be85bbbd8701d7981ea473d451611a70d384e"
Oct 30 04:43:11.752387 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:11.752347    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:43:12.990945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:12.990911    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"]
Oct 30 04:43:13.981925 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:13.981886    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"]
Oct 30 04:43:13.982824 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:13.982802    5072 cpu_manager.go:414] "containerMap: removing container" podUID="002b0525-c6d4-4305-bb59-e3e470aaaff3" containerName="revoke-keys"
Oct 30 04:43:13.982824 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:13.982824    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002b0525-c6d4-4305-bb59-e3e470aaaff3" containerName="revoke-keys"
Oct 30 04:43:13.982994 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:13.982985    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="002b0525-c6d4-4305-bb59-e3e470aaaff3" containerName="revoke-keys"
Oct 30 04:43:13.987248 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:13.987218    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.010467 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.010435    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"]
Oct 30 04:43:14.017742 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.017715    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-bbce72\"/\"auth-test-model-bbce72-kserve-self-signed-certs\""
Oct 30 04:43:14.017865 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.017838    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-bbce72\"/\"default-dockercfg-hmp9d\""
Oct 30 04:43:14.029389 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.029361    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghfg\" (UniqueName: \"kubernetes.io/projected/b837af6e-7029-42e9-85a0-14edfa6fbccd-kube-api-access-dghfg\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.029529 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.029431    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b837af6e-7029-42e9-85a0-14edfa6fbccd-tls-certs\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.029630 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.029606    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-dshm\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.029732 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.029714    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-tmp-dir\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.029796 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.029749    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-home\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.029895 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.029871    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-model-cache\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.058506 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.058478    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-bbce72\"/\"openshift-service-ca.crt\""
Oct 30 04:43:14.058506 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.058491    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-bbce72\"/\"kube-root-ca.crt\""
Oct 30 04:43:14.131751 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.131712    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-dshm\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.131939 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.131812    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-tmp-dir\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.131939 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.131846    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-home\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.131939 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.131901    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-model-cache\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.132146 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.131989    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghfg\" (UniqueName: \"kubernetes.io/projected/b837af6e-7029-42e9-85a0-14edfa6fbccd-kube-api-access-dghfg\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.132146 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.132070    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b837af6e-7029-42e9-85a0-14edfa6fbccd-tls-certs\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.132295 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.132269    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-tmp-dir\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.132753 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.132692    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-model-cache\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.132753 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.132722    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-home\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.134777 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.134750    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-dshm\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.134923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.134901    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b837af6e-7029-42e9-85a0-14edfa6fbccd-tls-certs\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.156347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.156309    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghfg\" (UniqueName: \"kubernetes.io/projected/b837af6e-7029-42e9-85a0-14edfa6fbccd-kube-api-access-dghfg\") pod \"auth-test-model-bbce72-kserve-58fc946fb9-7f46b\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.309911 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.309831    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:14.624131 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:14.624100    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"]
Oct 30 04:43:14.624376 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:43:14.624354    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb837af6e_7029_42e9_85a0_14edfa6fbccd.slice/crio-2cbded1ee1aa98c4ea666c658cf3a2b24a0fe9a1a04d5fa8464c64750852fe82 WatchSource:0}: Error finding container 2cbded1ee1aa98c4ea666c658cf3a2b24a0fe9a1a04d5fa8464c64750852fe82: Status 404 returned error can't find the container with id 2cbded1ee1aa98c4ea666c658cf3a2b24a0fe9a1a04d5fa8464c64750852fe82
Oct 30 04:43:15.456152 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:15.456119    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b" event={"ID":"b837af6e-7029-42e9-85a0-14edfa6fbccd","Type":"ContainerStarted","Data":"fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9"}
Oct 30 04:43:15.456152 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:15.456158    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b" event={"ID":"b837af6e-7029-42e9-85a0-14edfa6fbccd","Type":"ContainerStarted","Data":"2cbded1ee1aa98c4ea666c658cf3a2b24a0fe9a1a04d5fa8464c64750852fe82"}
Oct 30 04:43:15.501901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:15.501854    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b" podStartSLOduration=2.501841383 podStartE2EDuration="2.501841383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:15.499221197 +0000 UTC m=+2775.240644294" watchObservedRunningTime="2026-10-30 04:43:15.501841383 +0000 UTC m=+2775.243264480"
Oct 30 04:43:16.310141 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.310103    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:16.579427 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.579350    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"]
Oct 30 04:43:16.583526 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.583509    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:16.602797 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.602767    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"]
Oct 30 04:43:16.612093 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.611449    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:43:16.612093 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.611500    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:43:16.661999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.661964    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf30567-3169-4681-91a2-8185771bd570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c180b0f-cgpfx\" (UID: \"7bf30567-3169-4681-91a2-8185771bd570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:16.662154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.662138    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf30567-3169-4681-91a2-8185771bd570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c180b0f-cgpfx\" (UID: \"7bf30567-3169-4681-91a2-8185771bd570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:16.662200 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.662164    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5fjd\" (UniqueName: \"kubernetes.io/projected/7bf30567-3169-4681-91a2-8185771bd570-kube-api-access-w5fjd\") pod \"maas-api-revoke-sub-4c180b0f-cgpfx\" (UID: \"7bf30567-3169-4681-91a2-8185771bd570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:16.763115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.763091    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf30567-3169-4681-91a2-8185771bd570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c180b0f-cgpfx\" (UID: \"7bf30567-3169-4681-91a2-8185771bd570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:16.763115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.763121    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fjd\" (UniqueName: \"kubernetes.io/projected/7bf30567-3169-4681-91a2-8185771bd570-kube-api-access-w5fjd\") pod \"maas-api-revoke-sub-4c180b0f-cgpfx\" (UID: \"7bf30567-3169-4681-91a2-8185771bd570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:16.763268 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.763178    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf30567-3169-4681-91a2-8185771bd570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c180b0f-cgpfx\" (UID: \"7bf30567-3169-4681-91a2-8185771bd570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:16.763843 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.763827    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf30567-3169-4681-91a2-8185771bd570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c180b0f-cgpfx\" (UID: \"7bf30567-3169-4681-91a2-8185771bd570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:16.765687 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.765663    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf30567-3169-4681-91a2-8185771bd570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c180b0f-cgpfx\" (UID: \"7bf30567-3169-4681-91a2-8185771bd570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:16.787626 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.787596    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5fjd\" (UniqueName: \"kubernetes.io/projected/7bf30567-3169-4681-91a2-8185771bd570-kube-api-access-w5fjd\") pod \"maas-api-revoke-sub-4c180b0f-cgpfx\" (UID: \"7bf30567-3169-4681-91a2-8185771bd570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:16.899402 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:16.899340    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:17.200665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:17.200634    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"]
Oct 30 04:43:17.201027 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:43:17.201003    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf30567_3169_4681_91a2_8185771bd570.slice/crio-d1c454a75d24d50e42ff8d1a4edff56b00e95b9375e1de240edaef378b01f745 WatchSource:0}: Error finding container d1c454a75d24d50e42ff8d1a4edff56b00e95b9375e1de240edaef378b01f745: Status 404 returned error can't find the container with id d1c454a75d24d50e42ff8d1a4edff56b00e95b9375e1de240edaef378b01f745
Oct 30 04:43:17.469872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:17.469794    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx" event={"ID":"7bf30567-3169-4681-91a2-8185771bd570","Type":"ContainerStarted","Data":"d1c454a75d24d50e42ff8d1a4edff56b00e95b9375e1de240edaef378b01f745"}
Oct 30 04:43:17.991426 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:17.991373    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87" podUID="5bb9b08f-8c29-48ea-abe1-8ab303a46756" containerName="payload-processing" containerID="cri-o://be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4" gracePeriod=25
Oct 30 04:43:18.271148 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.271124    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"
Oct 30 04:43:18.382681 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.382649    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb9b08f-8c29-48ea-abe1-8ab303a46756-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb9b08f-8c29-48ea-abe1-8ab303a46756-plugins-config-volume\") pod \"5bb9b08f-8c29-48ea-abe1-8ab303a46756\" (UID: \"5bb9b08f-8c29-48ea-abe1-8ab303a46756\") "
Oct 30 04:43:18.382840 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.382708    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb9b08f-8c29-48ea-abe1-8ab303a46756-kube-api-access-9cc95\" (UniqueName: \"kubernetes.io/projected/5bb9b08f-8c29-48ea-abe1-8ab303a46756-kube-api-access-9cc95\") pod \"5bb9b08f-8c29-48ea-abe1-8ab303a46756\" (UID: \"5bb9b08f-8c29-48ea-abe1-8ab303a46756\") "
Oct 30 04:43:18.383044 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.383016    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb9b08f-8c29-48ea-abe1-8ab303a46756-plugins-config-volume" pod "5bb9b08f-8c29-48ea-abe1-8ab303a46756" (UID: "5bb9b08f-8c29-48ea-abe1-8ab303a46756"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:43:18.383267 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.383248    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb9b08f-8c29-48ea-abe1-8ab303a46756-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:18.384975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.384953    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb9b08f-8c29-48ea-abe1-8ab303a46756-kube-api-access-9cc95" pod "5bb9b08f-8c29-48ea-abe1-8ab303a46756" (UID: "5bb9b08f-8c29-48ea-abe1-8ab303a46756"). InnerVolumeSpecName "kube-api-access-9cc95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:18.478377 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.478332    5072 generic.go:356] "Generic (PLEG): container finished" podID="5bb9b08f-8c29-48ea-abe1-8ab303a46756" containerID="be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4" exitCode=0
Oct 30 04:43:18.478548 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.478447    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"
Oct 30 04:43:18.478548 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.478458    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87" event={"ID":"5bb9b08f-8c29-48ea-abe1-8ab303a46756","Type":"ContainerDied","Data":"be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4"}
Oct 30 04:43:18.478548 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.478529    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87" event={"ID":"5bb9b08f-8c29-48ea-abe1-8ab303a46756","Type":"ContainerDied","Data":"c0a6a3fe9492fdec1735154217447479a28ff9e3733b9104cf648284404819fb"}
Oct 30 04:43:18.478718 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.478557    5072 scope.go:122] "RemoveContainer" containerID="be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4"
Oct 30 04:43:18.480453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.480421    5072 generic.go:356] "Generic (PLEG): container finished" podID="7bf30567-3169-4681-91a2-8185771bd570" containerID="81f3fdf758c4a05499a7761c9c4d665b60806f62b8a4392cae05eaa6a7b240cf" exitCode=0
Oct 30 04:43:18.480559 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.480467    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx" event={"ID":"7bf30567-3169-4681-91a2-8185771bd570","Type":"ContainerDied","Data":"81f3fdf758c4a05499a7761c9c4d665b60806f62b8a4392cae05eaa6a7b240cf"}
Oct 30 04:43:18.484672 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.484650    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cc95\" (UniqueName: \"kubernetes.io/projected/5bb9b08f-8c29-48ea-abe1-8ab303a46756-kube-api-access-9cc95\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:18.489281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.489261    5072 scope.go:122] "RemoveContainer" containerID="be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4"
Oct 30 04:43:18.489546 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:18.489526    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4\": container with ID starting with be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4 not found: ID does not exist" containerID="be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4"
Oct 30 04:43:18.489600 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.489553    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4"} err="failed to get container status \"be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4\": rpc error: code = NotFound desc = could not find container \"be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4\": container with ID starting with be826250105f9cacb0bd9f9882ae676fcd0c613f3d52efe5976fe5b76fadfdd4 not found: ID does not exist"
Oct 30 04:43:18.576451 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.576370    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"]
Oct 30 04:43:18.580549 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:18.580524    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-20a650-7f7568947d-v6f87"]
Oct 30 04:43:19.630117 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:19.630070    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:19.697878 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:19.697847    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf30567-3169-4681-91a2-8185771bd570-kube-api-access-w5fjd\" (UniqueName: \"kubernetes.io/projected/7bf30567-3169-4681-91a2-8185771bd570-kube-api-access-w5fjd\") pod \"7bf30567-3169-4681-91a2-8185771bd570\" (UID: \"7bf30567-3169-4681-91a2-8185771bd570\") "
Oct 30 04:43:19.698037 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:19.697969    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf30567-3169-4681-91a2-8185771bd570-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf30567-3169-4681-91a2-8185771bd570-maas-api-openshift-ca\") pod \"7bf30567-3169-4681-91a2-8185771bd570\" (UID: \"7bf30567-3169-4681-91a2-8185771bd570\") "
Oct 30 04:43:19.698037 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:19.698032    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bf30567-3169-4681-91a2-8185771bd570-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf30567-3169-4681-91a2-8185771bd570-maas-api-xks-ca\") pod \"7bf30567-3169-4681-91a2-8185771bd570\" (UID: \"7bf30567-3169-4681-91a2-8185771bd570\") "
Oct 30 04:43:19.698371 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:19.698340    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf30567-3169-4681-91a2-8185771bd570-maas-api-openshift-ca" pod "7bf30567-3169-4681-91a2-8185771bd570" (UID: "7bf30567-3169-4681-91a2-8185771bd570"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:43:19.698514 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:19.698439    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf30567-3169-4681-91a2-8185771bd570-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:19.700332 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:19.700303    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf30567-3169-4681-91a2-8185771bd570-maas-api-xks-ca" pod "7bf30567-3169-4681-91a2-8185771bd570" (UID: "7bf30567-3169-4681-91a2-8185771bd570"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:43:19.700781 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:19.700760    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf30567-3169-4681-91a2-8185771bd570-kube-api-access-w5fjd" pod "7bf30567-3169-4681-91a2-8185771bd570" (UID: "7bf30567-3169-4681-91a2-8185771bd570"). InnerVolumeSpecName "kube-api-access-w5fjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:19.799586 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:19.799529    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5fjd\" (UniqueName: \"kubernetes.io/projected/7bf30567-3169-4681-91a2-8185771bd570-kube-api-access-w5fjd\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:19.799586 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:19.799554    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf30567-3169-4681-91a2-8185771bd570-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:20.409520 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:20.409479    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb9b08f-8c29-48ea-abe1-8ab303a46756" path="/var/lib/kubelet/pods/5bb9b08f-8c29-48ea-abe1-8ab303a46756/volumes"
Oct 30 04:43:20.494348 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:20.494322    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"
Oct 30 04:43:20.494502 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:20.494317    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx" event={"ID":"7bf30567-3169-4681-91a2-8185771bd570","Type":"ContainerDied","Data":"d1c454a75d24d50e42ff8d1a4edff56b00e95b9375e1de240edaef378b01f745"}
Oct 30 04:43:20.494502 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:20.494446    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1c454a75d24d50e42ff8d1a4edff56b00e95b9375e1de240edaef378b01f745"
Oct 30 04:43:24.310803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:24.310770    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:24.323572 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:24.323546    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:24.529495 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:24.529468    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:43:36.501210 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:36.501158    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"]
Oct 30 04:43:41.752645 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:41.752607    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:43:44.284339 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.284306    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"]
Oct 30 04:43:44.532659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.531907    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11258-0a4c2a"]
Oct 30 04:43:44.532659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.532616    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7bf30567-3169-4681-91a2-8185771bd570" containerName="revoke-keys"
Oct 30 04:43:44.532659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.532637    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf30567-3169-4681-91a2-8185771bd570" containerName="revoke-keys"
Oct 30 04:43:44.532659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.532655    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5bb9b08f-8c29-48ea-abe1-8ab303a46756" containerName="payload-processing"
Oct 30 04:43:44.532659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.532661    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb9b08f-8c29-48ea-abe1-8ab303a46756" containerName="payload-processing"
Oct 30 04:43:44.533097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.532796    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf30567-3169-4681-91a2-8185771bd570" containerName="revoke-keys"
Oct 30 04:43:44.533097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.532807    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb9b08f-8c29-48ea-abe1-8ab303a46756" containerName="payload-processing"
Oct 30 04:43:44.540593 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.540039    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-0a4c2a"
Oct 30 04:43:44.551115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.551067    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11258-0a4c2a"]
Oct 30 04:43:44.555306 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.555285    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gsgvm\""
Oct 30 04:43:44.598547 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.598514    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ds8f\" (UniqueName: \"kubernetes.io/projected/2e2ac94b-9d65-422f-b7a0-c2e43e836a24-kube-api-access-8ds8f\") pod \"test-curl-11258-0a4c2a\" (UID: \"2e2ac94b-9d65-422f-b7a0-c2e43e836a24\") " pod="opendatahub/test-curl-11258-0a4c2a"
Oct 30 04:43:44.699988 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.699958    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ds8f\" (UniqueName: \"kubernetes.io/projected/2e2ac94b-9d65-422f-b7a0-c2e43e836a24-kube-api-access-8ds8f\") pod \"test-curl-11258-0a4c2a\" (UID: \"2e2ac94b-9d65-422f-b7a0-c2e43e836a24\") " pod="opendatahub/test-curl-11258-0a4c2a"
Oct 30 04:43:44.724141 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.724116    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ds8f\" (UniqueName: \"kubernetes.io/projected/2e2ac94b-9d65-422f-b7a0-c2e43e836a24-kube-api-access-8ds8f\") pod \"test-curl-11258-0a4c2a\" (UID: \"2e2ac94b-9d65-422f-b7a0-c2e43e836a24\") " pod="opendatahub/test-curl-11258-0a4c2a"
Oct 30 04:43:44.854740 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:44.854664    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-0a4c2a"
Oct 30 04:43:45.160097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:45.159904    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11258-0a4c2a"]
Oct 30 04:43:45.160215 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:43:45.160145    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e2ac94b_9d65_422f_b7a0_c2e43e836a24.slice/crio-10dc8b88a4fa673c37cc84aaed68f9ef14f2c524b84e9531c32db7f5339e117a WatchSource:0}: Error finding container 10dc8b88a4fa673c37cc84aaed68f9ef14f2c524b84e9531c32db7f5339e117a: Status 404 returned error can't find the container with id 10dc8b88a4fa673c37cc84aaed68f9ef14f2c524b84e9531c32db7f5339e117a
Oct 30 04:43:45.651776 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:45.651741    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11258-0a4c2a" event={"ID":"2e2ac94b-9d65-422f-b7a0-c2e43e836a24","Type":"ContainerStarted","Data":"e09c46f53af5c9a24ca7e610b484728ef4e6a0d899066ed8ecff8f38a99fbbcd"}
Oct 30 04:43:45.652218 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:45.651791    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11258-0a4c2a" event={"ID":"2e2ac94b-9d65-422f-b7a0-c2e43e836a24","Type":"ContainerStarted","Data":"10dc8b88a4fa673c37cc84aaed68f9ef14f2c524b84e9531c32db7f5339e117a"}
Oct 30 04:43:45.691308 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:45.691256    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11258-0a4c2a" podStartSLOduration=1.69124302 podStartE2EDuration="1.69124302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:45.690384597 +0000 UTC m=+2805.431807694" watchObservedRunningTime="2026-10-30 04:43:45.69124302 +0000 UTC m=+2805.432666161"
Oct 30 04:43:46.582059 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:46.582028    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11258-0a4c2a"]
Oct 30 04:43:46.589297 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:46.589271    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11258-0a4c2a"]
Oct 30 04:43:46.791241 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:46.791212    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11258-39b691"]
Oct 30 04:43:46.791810 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:46.791795    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2e2ac94b-9d65-422f-b7a0-c2e43e836a24" containerName="test-curl-11258-0a4c2a"
Oct 30 04:43:46.791853 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:46.791811    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2ac94b-9d65-422f-b7a0-c2e43e836a24" containerName="test-curl-11258-0a4c2a"
Oct 30 04:43:46.791924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:46.791914    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2ac94b-9d65-422f-b7a0-c2e43e836a24" containerName="test-curl-11258-0a4c2a"
Oct 30 04:43:46.795423 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:46.795397    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-39b691"
Oct 30 04:43:46.811227 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:46.811197    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11258-39b691"]
Oct 30 04:43:46.822731 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:46.822707    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvtpw\" (UniqueName: \"kubernetes.io/projected/60dad8ae-7b38-449d-ab5e-a02aca1fc7f9-kube-api-access-gvtpw\") pod \"test-curl-11258-39b691\" (UID: \"60dad8ae-7b38-449d-ab5e-a02aca1fc7f9\") " pod="opendatahub/test-curl-11258-39b691"
Oct 30 04:43:46.929360 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:46.929285    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvtpw\" (UniqueName: \"kubernetes.io/projected/60dad8ae-7b38-449d-ab5e-a02aca1fc7f9-kube-api-access-gvtpw\") pod \"test-curl-11258-39b691\" (UID: \"60dad8ae-7b38-449d-ab5e-a02aca1fc7f9\") " pod="opendatahub/test-curl-11258-39b691"
Oct 30 04:43:46.953185 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:46.953158    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvtpw\" (UniqueName: \"kubernetes.io/projected/60dad8ae-7b38-449d-ab5e-a02aca1fc7f9-kube-api-access-gvtpw\") pod \"test-curl-11258-39b691\" (UID: \"60dad8ae-7b38-449d-ab5e-a02aca1fc7f9\") " pod="opendatahub/test-curl-11258-39b691"
Oct 30 04:43:47.109450 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:47.109416    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-39b691"
Oct 30 04:43:47.414683 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:43:47.414648    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60dad8ae_7b38_449d_ab5e_a02aca1fc7f9.slice/crio-02785d6c4de40f249f455febcd2045c99dcb5aa28b6c957d0f172775308d7158 WatchSource:0}: Error finding container 02785d6c4de40f249f455febcd2045c99dcb5aa28b6c957d0f172775308d7158: Status 404 returned error can't find the container with id 02785d6c4de40f249f455febcd2045c99dcb5aa28b6c957d0f172775308d7158
Oct 30 04:43:47.416029 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:47.415998    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11258-39b691"]
Oct 30 04:43:47.667863 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:47.667763    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11258-39b691" event={"ID":"60dad8ae-7b38-449d-ab5e-a02aca1fc7f9","Type":"ContainerStarted","Data":"758097d19b1b9e5d1bb30c872ee7718e8fe36fd4d7f44a6c54be602c98536231"}
Oct 30 04:43:47.667863 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:47.667808    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11258-39b691" event={"ID":"60dad8ae-7b38-449d-ab5e-a02aca1fc7f9","Type":"ContainerStarted","Data":"02785d6c4de40f249f455febcd2045c99dcb5aa28b6c957d0f172775308d7158"}
Oct 30 04:43:47.668159 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:47.667915    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11258-0a4c2a" podUID="2e2ac94b-9d65-422f-b7a0-c2e43e836a24" containerName="test-curl-11258-0a4c2a" containerID="cri-o://e09c46f53af5c9a24ca7e610b484728ef4e6a0d899066ed8ecff8f38a99fbbcd" gracePeriod=2
Oct 30 04:43:47.681941 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:47.681907    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-0a4c2a\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="2e2ac94b-9d65-422f-b7a0-c2e43e836a24" pod="opendatahub/test-curl-11258-0a4c2a"
Oct 30 04:43:47.722915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:47.722869    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11258-39b691" podStartSLOduration=1.7228562269999999 podStartE2EDuration="1.722856227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:47.720121032 +0000 UTC m=+2807.461544130" watchObservedRunningTime="2026-10-30 04:43:47.722856227 +0000 UTC m=+2807.464279325"
Oct 30 04:43:47.734339 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:47.734313    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-0a4c2a\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="2e2ac94b-9d65-422f-b7a0-c2e43e836a24" pod="opendatahub/test-curl-11258-0a4c2a"
Oct 30 04:43:48.552794 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:48.552765    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11258-39b691"]
Oct 30 04:43:48.558592 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:48.558566    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11258-39b691"]
Oct 30 04:43:48.713172 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:48.713140    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11258-b9778e"]
Oct 30 04:43:48.713697 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:48.713684    5072 cpu_manager.go:414] "containerMap: removing container" podUID="60dad8ae-7b38-449d-ab5e-a02aca1fc7f9" containerName="test-curl-11258-39b691"
Oct 30 04:43:48.713748 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:48.713699    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60dad8ae-7b38-449d-ab5e-a02aca1fc7f9" containerName="test-curl-11258-39b691"
Oct 30 04:43:48.713830 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:48.713821    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="60dad8ae-7b38-449d-ab5e-a02aca1fc7f9" containerName="test-curl-11258-39b691"
Oct 30 04:43:48.717336 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:48.717319    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-b9778e"
Oct 30 04:43:48.733550 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:48.733526    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11258-b9778e"]
Oct 30 04:43:48.750923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:48.750895    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b528g\" (UniqueName: \"kubernetes.io/projected/38a34c45-9623-47c6-8dc1-24238447e50d-kube-api-access-b528g\") pod \"test-curl-11258-b9778e\" (UID: \"38a34c45-9623-47c6-8dc1-24238447e50d\") " pod="opendatahub/test-curl-11258-b9778e"
Oct 30 04:43:48.852581 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:48.852497    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b528g\" (UniqueName: \"kubernetes.io/projected/38a34c45-9623-47c6-8dc1-24238447e50d-kube-api-access-b528g\") pod \"test-curl-11258-b9778e\" (UID: \"38a34c45-9623-47c6-8dc1-24238447e50d\") " pod="opendatahub/test-curl-11258-b9778e"
Oct 30 04:43:48.876188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:48.876162    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b528g\" (UniqueName: \"kubernetes.io/projected/38a34c45-9623-47c6-8dc1-24238447e50d-kube-api-access-b528g\") pod \"test-curl-11258-b9778e\" (UID: \"38a34c45-9623-47c6-8dc1-24238447e50d\") " pod="opendatahub/test-curl-11258-b9778e"
Oct 30 04:43:49.029878 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:49.029847    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-b9778e"
Oct 30 04:43:49.330416 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:49.330381    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11258-b9778e"]
Oct 30 04:43:49.330612 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:43:49.330590    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38a34c45_9623_47c6_8dc1_24238447e50d.slice/crio-3bb091415839bfc2bcfa408d3d14f4acfff0767fcd631eae339358c4864c66da WatchSource:0}: Error finding container 3bb091415839bfc2bcfa408d3d14f4acfff0767fcd631eae339358c4864c66da: Status 404 returned error can't find the container with id 3bb091415839bfc2bcfa408d3d14f4acfff0767fcd631eae339358c4864c66da
Oct 30 04:43:49.682065 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:49.682030    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11258-b9778e" event={"ID":"38a34c45-9623-47c6-8dc1-24238447e50d","Type":"ContainerStarted","Data":"fbb7b7f5e0514903b5a4cd7c8a0f8aa83b1fe6eac19a1aec3ad0ce79997708a2"}
Oct 30 04:43:49.682534 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:49.682071    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11258-39b691" podUID="60dad8ae-7b38-449d-ab5e-a02aca1fc7f9" containerName="test-curl-11258-39b691" containerID="cri-o://758097d19b1b9e5d1bb30c872ee7718e8fe36fd4d7f44a6c54be602c98536231" gracePeriod=2
Oct 30 04:43:49.682534 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:49.682235    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11258-b9778e" event={"ID":"38a34c45-9623-47c6-8dc1-24238447e50d","Type":"ContainerStarted","Data":"3bb091415839bfc2bcfa408d3d14f4acfff0767fcd631eae339358c4864c66da"}
Oct 30 04:43:49.721900 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:49.721851    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11258-b9778e" podStartSLOduration=1.7218356940000001 podStartE2EDuration="1.721835694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:49.720533536 +0000 UTC m=+2809.461956630" watchObservedRunningTime="2026-10-30 04:43:49.721835694 +0000 UTC m=+2809.463258791"
Oct 30 04:43:49.734587 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:49.734557    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-39b691\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="60dad8ae-7b38-449d-ab5e-a02aca1fc7f9" pod="opendatahub/test-curl-11258-39b691"
Oct 30 04:43:49.748863 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:49.748822    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-39b691\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="60dad8ae-7b38-449d-ab5e-a02aca1fc7f9" pod="opendatahub/test-curl-11258-39b691"
Oct 30 04:43:49.867476 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:49.867452    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-0a4c2a"
Oct 30 04:43:49.881657 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:49.881627    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-39b691\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="60dad8ae-7b38-449d-ab5e-a02aca1fc7f9" pod="opendatahub/test-curl-11258-39b691"
Oct 30 04:43:49.965686 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:49.965658    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2ac94b-9d65-422f-b7a0-c2e43e836a24-kube-api-access-8ds8f\" (UniqueName: \"kubernetes.io/projected/2e2ac94b-9d65-422f-b7a0-c2e43e836a24-kube-api-access-8ds8f\") pod \"2e2ac94b-9d65-422f-b7a0-c2e43e836a24\" (UID: \"2e2ac94b-9d65-422f-b7a0-c2e43e836a24\") "
Oct 30 04:43:49.968119 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:49.968055    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2ac94b-9d65-422f-b7a0-c2e43e836a24-kube-api-access-8ds8f" pod "2e2ac94b-9d65-422f-b7a0-c2e43e836a24" (UID: "2e2ac94b-9d65-422f-b7a0-c2e43e836a24"). InnerVolumeSpecName "kube-api-access-8ds8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:50.067125 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.067090    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ds8f\" (UniqueName: \"kubernetes.io/projected/2e2ac94b-9d65-422f-b7a0-c2e43e836a24-kube-api-access-8ds8f\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:50.408312 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.408239    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2ac94b-9d65-422f-b7a0-c2e43e836a24" path="/var/lib/kubelet/pods/2e2ac94b-9d65-422f-b7a0-c2e43e836a24/volumes"
Oct 30 04:43:50.417623 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:50.417594    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-39b691\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="60dad8ae-7b38-449d-ab5e-a02aca1fc7f9" pod="opendatahub/test-curl-11258-39b691"
Oct 30 04:43:50.596197 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.596164    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11258-b9778e"]
Oct 30 04:43:50.602998 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.602966    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11258-b9778e"]
Oct 30 04:43:50.689972 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.689941    5072 generic.go:356] "Generic (PLEG): container finished" podID="2e2ac94b-9d65-422f-b7a0-c2e43e836a24" containerID="e09c46f53af5c9a24ca7e610b484728ef4e6a0d899066ed8ecff8f38a99fbbcd" exitCode=137
Oct 30 04:43:50.690471 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.689987    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-0a4c2a"
Oct 30 04:43:50.690471 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.690035    5072 scope.go:122] "RemoveContainer" containerID="e09c46f53af5c9a24ca7e610b484728ef4e6a0d899066ed8ecff8f38a99fbbcd"
Oct 30 04:43:50.699972 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.699948    5072 scope.go:122] "RemoveContainer" containerID="e09c46f53af5c9a24ca7e610b484728ef4e6a0d899066ed8ecff8f38a99fbbcd"
Oct 30 04:43:50.700293 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:50.700274    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09c46f53af5c9a24ca7e610b484728ef4e6a0d899066ed8ecff8f38a99fbbcd\": container with ID starting with e09c46f53af5c9a24ca7e610b484728ef4e6a0d899066ed8ecff8f38a99fbbcd not found: ID does not exist" containerID="e09c46f53af5c9a24ca7e610b484728ef4e6a0d899066ed8ecff8f38a99fbbcd"
Oct 30 04:43:50.700357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.700302    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09c46f53af5c9a24ca7e610b484728ef4e6a0d899066ed8ecff8f38a99fbbcd"} err="failed to get container status \"e09c46f53af5c9a24ca7e610b484728ef4e6a0d899066ed8ecff8f38a99fbbcd\": rpc error: code = NotFound desc = could not find container \"e09c46f53af5c9a24ca7e610b484728ef4e6a0d899066ed8ecff8f38a99fbbcd\": container with ID starting with e09c46f53af5c9a24ca7e610b484728ef4e6a0d899066ed8ecff8f38a99fbbcd not found: ID does not exist"
Oct 30 04:43:50.752066 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.752030    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11258-0cd42e"]
Oct 30 04:43:50.752703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.752682    5072 cpu_manager.go:414] "containerMap: removing container" podUID="38a34c45-9623-47c6-8dc1-24238447e50d" containerName="test-curl-11258-b9778e"
Oct 30 04:43:50.752703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.752699    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a34c45-9623-47c6-8dc1-24238447e50d" containerName="test-curl-11258-b9778e"
Oct 30 04:43:50.752860 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.752807    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a34c45-9623-47c6-8dc1-24238447e50d" containerName="test-curl-11258-b9778e"
Oct 30 04:43:50.757397 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.757366    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-0cd42e"
Oct 30 04:43:50.787397 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.787365    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11258-0cd42e"]
Oct 30 04:43:50.885318 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.885283    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrjf6\" (UniqueName: \"kubernetes.io/projected/5b242228-c33a-487f-9021-c307c5c2ce21-kube-api-access-nrjf6\") pod \"test-curl-11258-0cd42e\" (UID: \"5b242228-c33a-487f-9021-c307c5c2ce21\") " pod="opendatahub/test-curl-11258-0cd42e"
Oct 30 04:43:50.986247 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:50.986148    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjf6\" (UniqueName: \"kubernetes.io/projected/5b242228-c33a-487f-9021-c307c5c2ce21-kube-api-access-nrjf6\") pod \"test-curl-11258-0cd42e\" (UID: \"5b242228-c33a-487f-9021-c307c5c2ce21\") " pod="opendatahub/test-curl-11258-0cd42e"
Oct 30 04:43:51.009914 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.009885    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjf6\" (UniqueName: \"kubernetes.io/projected/5b242228-c33a-487f-9021-c307c5c2ce21-kube-api-access-nrjf6\") pod \"test-curl-11258-0cd42e\" (UID: \"5b242228-c33a-487f-9021-c307c5c2ce21\") " pod="opendatahub/test-curl-11258-0cd42e"
Oct 30 04:43:51.070608 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.070576    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-0cd42e"
Oct 30 04:43:51.381324 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.381275    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11258-0cd42e"]
Oct 30 04:43:51.384168 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:43:51.384121    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b242228_c33a_487f_9021_c307c5c2ce21.slice/crio-92f6ca9abab30bb81e83774e0dd8c8e2c88392713ccd74db0fb48c296b3ddfa0 WatchSource:0}: Error finding container 92f6ca9abab30bb81e83774e0dd8c8e2c88392713ccd74db0fb48c296b3ddfa0: Status 404 returned error can't find the container with id 92f6ca9abab30bb81e83774e0dd8c8e2c88392713ccd74db0fb48c296b3ddfa0
Oct 30 04:43:51.524468 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.524421    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f" podUID="f6abc515-914d-4443-9237-8b677d1594c1" containerName="main" containerID="cri-o://8f661381f1fe66bd09853307e9fc0703e91218df72b65d22dd20060dbc90aa02" gracePeriod=45
Oct 30 04:43:51.697630 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.697594    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11258-0cd42e" event={"ID":"5b242228-c33a-487f-9021-c307c5c2ce21","Type":"ContainerStarted","Data":"337cd3b2129743236705daacd15115fd078679a5667689b589ff54604d4f384d"}
Oct 30 04:43:51.698040 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.697633    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11258-0cd42e" event={"ID":"5b242228-c33a-487f-9021-c307c5c2ce21","Type":"ContainerStarted","Data":"92f6ca9abab30bb81e83774e0dd8c8e2c88392713ccd74db0fb48c296b3ddfa0"}
Oct 30 04:43:51.700504 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.700480    5072 generic.go:356] "Generic (PLEG): container finished" podID="f6abc515-914d-4443-9237-8b677d1594c1" containerID="8f661381f1fe66bd09853307e9fc0703e91218df72b65d22dd20060dbc90aa02" exitCode=0
Oct 30 04:43:51.700709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.700558    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f" event={"ID":"f6abc515-914d-4443-9237-8b677d1594c1","Type":"ContainerDied","Data":"8f661381f1fe66bd09853307e9fc0703e91218df72b65d22dd20060dbc90aa02"}
Oct 30 04:43:51.700709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.700633    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11258-b9778e" podUID="38a34c45-9623-47c6-8dc1-24238447e50d" containerName="test-curl-11258-b9778e" containerID="cri-o://fbb7b7f5e0514903b5a4cd7c8a0f8aa83b1fe6eac19a1aec3ad0ce79997708a2" gracePeriod=2
Oct 30 04:43:51.742357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.742298    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11258-0cd42e" podStartSLOduration=1.742277919 podStartE2EDuration="1.742277919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:51.738218054 +0000 UTC m=+2811.479641154" watchObservedRunningTime="2026-10-30 04:43:51.742277919 +0000 UTC m=+2811.483701017"
Oct 30 04:43:51.753240 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:51.753201    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-b9778e\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="38a34c45-9623-47c6-8dc1-24238447e50d" pod="opendatahub/test-curl-11258-b9778e"
Oct 30 04:43:51.798501 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.798477    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:43:51.861572 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:51.860212    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-b9778e\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="38a34c45-9623-47c6-8dc1-24238447e50d" pod="opendatahub/test-curl-11258-b9778e"
Oct 30 04:43:51.897807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.897775    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-model-cache\") pod \"f6abc515-914d-4443-9237-8b677d1594c1\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") "
Oct 30 04:43:51.897967 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.897886    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-tmp-dir\") pod \"f6abc515-914d-4443-9237-8b677d1594c1\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") "
Oct 30 04:43:51.897967 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.897939    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6abc515-914d-4443-9237-8b677d1594c1-kube-api-access-x8g58\" (UniqueName: \"kubernetes.io/projected/f6abc515-914d-4443-9237-8b677d1594c1-kube-api-access-x8g58\") pod \"f6abc515-914d-4443-9237-8b677d1594c1\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") "
Oct 30 04:43:51.898154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.897979    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-home\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-home\") pod \"f6abc515-914d-4443-9237-8b677d1594c1\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") "
Oct 30 04:43:51.898154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.898003    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-model-cache" pod "f6abc515-914d-4443-9237-8b677d1594c1" (UID: "f6abc515-914d-4443-9237-8b677d1594c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:51.898154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.898014    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-dshm\") pod \"f6abc515-914d-4443-9237-8b677d1594c1\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") "
Oct 30 04:43:51.898154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.898130    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6abc515-914d-4443-9237-8b677d1594c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6abc515-914d-4443-9237-8b677d1594c1-tls-certs\") pod \"f6abc515-914d-4443-9237-8b677d1594c1\" (UID: \"f6abc515-914d-4443-9237-8b677d1594c1\") "
Oct 30 04:43:51.898393 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.898218    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-tmp-dir" pod "f6abc515-914d-4443-9237-8b677d1594c1" (UID: "f6abc515-914d-4443-9237-8b677d1594c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:51.898955 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.898633    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-home" pod "f6abc515-914d-4443-9237-8b677d1594c1" (UID: "f6abc515-914d-4443-9237-8b677d1594c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:51.898955 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.898659    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:51.898955 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.898680    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:51.900604 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.900584    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6abc515-914d-4443-9237-8b677d1594c1-kube-api-access-x8g58" pod "f6abc515-914d-4443-9237-8b677d1594c1" (UID: "f6abc515-914d-4443-9237-8b677d1594c1"). InnerVolumeSpecName "kube-api-access-x8g58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:51.901004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.900976    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-dshm" pod "f6abc515-914d-4443-9237-8b677d1594c1" (UID: "f6abc515-914d-4443-9237-8b677d1594c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:51.901004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.900982    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6abc515-914d-4443-9237-8b677d1594c1-tls-certs" pod "f6abc515-914d-4443-9237-8b677d1594c1" (UID: "f6abc515-914d-4443-9237-8b677d1594c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:43:51.902287 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.902268    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-39b691"
Oct 30 04:43:51.916342 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:51.916316    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-b9778e\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="38a34c45-9623-47c6-8dc1-24238447e50d" pod="opendatahub/test-curl-11258-b9778e"
Oct 30 04:43:52.000010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:51.999978    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60dad8ae-7b38-449d-ab5e-a02aca1fc7f9-kube-api-access-gvtpw\" (UniqueName: \"kubernetes.io/projected/60dad8ae-7b38-449d-ab5e-a02aca1fc7f9-kube-api-access-gvtpw\") pod \"60dad8ae-7b38-449d-ab5e-a02aca1fc7f9\" (UID: \"60dad8ae-7b38-449d-ab5e-a02aca1fc7f9\") "
Oct 30 04:43:52.000299 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.000285    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:52.000351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.000300    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6abc515-914d-4443-9237-8b677d1594c1-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:52.000351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.000309    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6abc515-914d-4443-9237-8b677d1594c1-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:52.000351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.000318    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8g58\" (UniqueName: \"kubernetes.io/projected/f6abc515-914d-4443-9237-8b677d1594c1-kube-api-access-x8g58\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:52.002259 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.002235    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60dad8ae-7b38-449d-ab5e-a02aca1fc7f9-kube-api-access-gvtpw" pod "60dad8ae-7b38-449d-ab5e-a02aca1fc7f9" (UID: "60dad8ae-7b38-449d-ab5e-a02aca1fc7f9"). InnerVolumeSpecName "kube-api-access-gvtpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:52.101707 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.101673    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvtpw\" (UniqueName: \"kubernetes.io/projected/60dad8ae-7b38-449d-ab5e-a02aca1fc7f9-kube-api-access-gvtpw\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:52.409584 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.409533    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60dad8ae-7b38-449d-ab5e-a02aca1fc7f9" path="/var/lib/kubelet/pods/60dad8ae-7b38-449d-ab5e-a02aca1fc7f9/volumes"
Oct 30 04:43:52.600353 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.600321    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11258-0cd42e"]
Oct 30 04:43:52.602873 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.602843    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11258-0cd42e"]
Oct 30 04:43:52.707670 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.707639    5072 generic.go:356] "Generic (PLEG): container finished" podID="60dad8ae-7b38-449d-ab5e-a02aca1fc7f9" containerID="758097d19b1b9e5d1bb30c872ee7718e8fe36fd4d7f44a6c54be602c98536231" exitCode=137
Oct 30 04:43:52.708118 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.707692    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-39b691"
Oct 30 04:43:52.708118 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.707777    5072 scope.go:122] "RemoveContainer" containerID="758097d19b1b9e5d1bb30c872ee7718e8fe36fd4d7f44a6c54be602c98536231"
Oct 30 04:43:52.709336 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.709312    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"
Oct 30 04:43:52.709431 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.709367    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f" event={"ID":"f6abc515-914d-4443-9237-8b677d1594c1","Type":"ContainerDied","Data":"d51cec534f51252e3395e8b7e88c00f3ec76fbb5a851d8847278b8f3527b1518"}
Oct 30 04:43:52.717988 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.717970    5072 scope.go:122] "RemoveContainer" containerID="758097d19b1b9e5d1bb30c872ee7718e8fe36fd4d7f44a6c54be602c98536231"
Oct 30 04:43:52.718286 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:52.718267    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758097d19b1b9e5d1bb30c872ee7718e8fe36fd4d7f44a6c54be602c98536231\": container with ID starting with 758097d19b1b9e5d1bb30c872ee7718e8fe36fd4d7f44a6c54be602c98536231 not found: ID does not exist" containerID="758097d19b1b9e5d1bb30c872ee7718e8fe36fd4d7f44a6c54be602c98536231"
Oct 30 04:43:52.718372 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.718297    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758097d19b1b9e5d1bb30c872ee7718e8fe36fd4d7f44a6c54be602c98536231"} err="failed to get container status \"758097d19b1b9e5d1bb30c872ee7718e8fe36fd4d7f44a6c54be602c98536231\": rpc error: code = NotFound desc = could not find container \"758097d19b1b9e5d1bb30c872ee7718e8fe36fd4d7f44a6c54be602c98536231\": container with ID starting with 758097d19b1b9e5d1bb30c872ee7718e8fe36fd4d7f44a6c54be602c98536231 not found: ID does not exist"
Oct 30 04:43:52.718372 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.718320    5072 scope.go:122] "RemoveContainer" containerID="8f661381f1fe66bd09853307e9fc0703e91218df72b65d22dd20060dbc90aa02"
Oct 30 04:43:52.762483 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.762447    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"]
Oct 30 04:43:52.765679 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.765652    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11258-bdc934"]
Oct 30 04:43:52.766264 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.766249    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f6abc515-914d-4443-9237-8b677d1594c1" containerName="main"
Oct 30 04:43:52.766315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.766266    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6abc515-914d-4443-9237-8b677d1594c1" containerName="main"
Oct 30 04:43:52.766315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.766295    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5b242228-c33a-487f-9021-c307c5c2ce21" containerName="test-curl-11258-0cd42e"
Oct 30 04:43:52.766315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.766300    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b242228-c33a-487f-9021-c307c5c2ce21" containerName="test-curl-11258-0cd42e"
Oct 30 04:43:52.766430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.766410    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b242228-c33a-487f-9021-c307c5c2ce21" containerName="test-curl-11258-0cd42e"
Oct 30 04:43:52.766430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.766426    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6abc515-914d-4443-9237-8b677d1594c1" containerName="main"
Oct 30 04:43:52.770960 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.770943    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-bdc934"
Oct 30 04:43:52.776007 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.775982    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-fcd6a6/auth-test-model-fcd6a6-kserve-55b6c78bfc-kr75f"]
Oct 30 04:43:52.782210 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.782188    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11258-bdc934"]
Oct 30 04:43:52.910531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:52.910497    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z88md\" (UniqueName: \"kubernetes.io/projected/ba3cba4f-9d4c-4d0a-878e-795a7f947244-kube-api-access-z88md\") pod \"test-curl-11258-bdc934\" (UID: \"ba3cba4f-9d4c-4d0a-878e-795a7f947244\") " pod="opendatahub/test-curl-11258-bdc934"
Oct 30 04:43:53.011700 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:53.011624    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z88md\" (UniqueName: \"kubernetes.io/projected/ba3cba4f-9d4c-4d0a-878e-795a7f947244-kube-api-access-z88md\") pod \"test-curl-11258-bdc934\" (UID: \"ba3cba4f-9d4c-4d0a-878e-795a7f947244\") " pod="opendatahub/test-curl-11258-bdc934"
Oct 30 04:43:53.035585 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:53.035555    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z88md\" (UniqueName: \"kubernetes.io/projected/ba3cba4f-9d4c-4d0a-878e-795a7f947244-kube-api-access-z88md\") pod \"test-curl-11258-bdc934\" (UID: \"ba3cba4f-9d4c-4d0a-878e-795a7f947244\") " pod="opendatahub/test-curl-11258-bdc934"
Oct 30 04:43:53.085812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:53.085787    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-bdc934"
Oct 30 04:43:53.387360 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:53.387328    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11258-bdc934"]
Oct 30 04:43:53.387683 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:43:53.387658    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba3cba4f_9d4c_4d0a_878e_795a7f947244.slice/crio-d3509ca0cab0bb073d2505238ba5f0a37bd8604e865df6a153df7ec4b6e1ad8b WatchSource:0}: Error finding container d3509ca0cab0bb073d2505238ba5f0a37bd8604e865df6a153df7ec4b6e1ad8b: Status 404 returned error can't find the container with id d3509ca0cab0bb073d2505238ba5f0a37bd8604e865df6a153df7ec4b6e1ad8b
Oct 30 04:43:53.723127 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:53.723093    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11258-bdc934" event={"ID":"ba3cba4f-9d4c-4d0a-878e-795a7f947244","Type":"ContainerStarted","Data":"92cc330f7714d4881734e43633b0b2097f859d6aa923699d93f81513c918bbf7"}
Oct 30 04:43:53.723495 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:53.723144    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11258-bdc934" event={"ID":"ba3cba4f-9d4c-4d0a-878e-795a7f947244","Type":"ContainerStarted","Data":"d3509ca0cab0bb073d2505238ba5f0a37bd8604e865df6a153df7ec4b6e1ad8b"}
Oct 30 04:43:53.723495 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:53.723148    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11258-0cd42e" podUID="5b242228-c33a-487f-9021-c307c5c2ce21" containerName="test-curl-11258-0cd42e" containerID="cri-o://337cd3b2129743236705daacd15115fd078679a5667689b589ff54604d4f384d" gracePeriod=2
Oct 30 04:43:53.765361 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:53.765315    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11258-bdc934" podStartSLOduration=1.7653013450000001 podStartE2EDuration="1.765301345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:53.761906104 +0000 UTC m=+2813.503329201" watchObservedRunningTime="2026-10-30 04:43:53.765301345 +0000 UTC m=+2813.506724443"
Oct 30 04:43:53.775960 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:53.775928    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-0cd42e\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="5b242228-c33a-487f-9021-c307c5c2ce21" pod="opendatahub/test-curl-11258-0cd42e"
Oct 30 04:43:53.790139 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:53.790107    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-0cd42e\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="5b242228-c33a-487f-9021-c307c5c2ce21" pod="opendatahub/test-curl-11258-0cd42e"
Oct 30 04:43:53.898141 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:53.898118    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-b9778e"
Oct 30 04:43:53.912494 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:53.912464    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-0cd42e\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="5b242228-c33a-487f-9021-c307c5c2ce21" pod="opendatahub/test-curl-11258-0cd42e"
Oct 30 04:43:54.023776 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:54.023702    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a34c45-9623-47c6-8dc1-24238447e50d-kube-api-access-b528g\" (UniqueName: \"kubernetes.io/projected/38a34c45-9623-47c6-8dc1-24238447e50d-kube-api-access-b528g\") pod \"38a34c45-9623-47c6-8dc1-24238447e50d\" (UID: \"38a34c45-9623-47c6-8dc1-24238447e50d\") "
Oct 30 04:43:54.025955 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:54.025935    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a34c45-9623-47c6-8dc1-24238447e50d-kube-api-access-b528g" pod "38a34c45-9623-47c6-8dc1-24238447e50d" (UID: "38a34c45-9623-47c6-8dc1-24238447e50d"). InnerVolumeSpecName "kube-api-access-b528g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:54.125086 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:54.125053    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b528g\" (UniqueName: \"kubernetes.io/projected/38a34c45-9623-47c6-8dc1-24238447e50d-kube-api-access-b528g\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:54.408641 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:54.408566    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a34c45-9623-47c6-8dc1-24238447e50d" path="/var/lib/kubelet/pods/38a34c45-9623-47c6-8dc1-24238447e50d/volumes"
Oct 30 04:43:54.409020 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:54.409007    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6abc515-914d-4443-9237-8b677d1594c1" path="/var/lib/kubelet/pods/f6abc515-914d-4443-9237-8b677d1594c1/volumes"
Oct 30 04:43:54.637673 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:54.637639    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11258-bdc934"]
Oct 30 04:43:54.641574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:54.641544    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11258-bdc934"]
Oct 30 04:43:54.731423 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:54.731388    5072 generic.go:356] "Generic (PLEG): container finished" podID="38a34c45-9623-47c6-8dc1-24238447e50d" containerID="fbb7b7f5e0514903b5a4cd7c8a0f8aa83b1fe6eac19a1aec3ad0ce79997708a2" exitCode=137
Oct 30 04:43:54.731933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:54.731459    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-b9778e"
Oct 30 04:43:54.731933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:54.731488    5072 scope.go:122] "RemoveContainer" containerID="fbb7b7f5e0514903b5a4cd7c8a0f8aa83b1fe6eac19a1aec3ad0ce79997708a2"
Oct 30 04:43:54.741281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:54.741261    5072 scope.go:122] "RemoveContainer" containerID="fbb7b7f5e0514903b5a4cd7c8a0f8aa83b1fe6eac19a1aec3ad0ce79997708a2"
Oct 30 04:43:54.741660 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:54.741629    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb7b7f5e0514903b5a4cd7c8a0f8aa83b1fe6eac19a1aec3ad0ce79997708a2\": container with ID starting with fbb7b7f5e0514903b5a4cd7c8a0f8aa83b1fe6eac19a1aec3ad0ce79997708a2 not found: ID does not exist" containerID="fbb7b7f5e0514903b5a4cd7c8a0f8aa83b1fe6eac19a1aec3ad0ce79997708a2"
Oct 30 04:43:54.741722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:54.741672    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb7b7f5e0514903b5a4cd7c8a0f8aa83b1fe6eac19a1aec3ad0ce79997708a2"} err="failed to get container status \"fbb7b7f5e0514903b5a4cd7c8a0f8aa83b1fe6eac19a1aec3ad0ce79997708a2\": rpc error: code = NotFound desc = could not find container \"fbb7b7f5e0514903b5a4cd7c8a0f8aa83b1fe6eac19a1aec3ad0ce79997708a2\": container with ID starting with fbb7b7f5e0514903b5a4cd7c8a0f8aa83b1fe6eac19a1aec3ad0ce79997708a2 not found: ID does not exist"
Oct 30 04:43:55.738992 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:55.738962    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11258-bdc934" podUID="ba3cba4f-9d4c-4d0a-878e-795a7f947244" containerName="test-curl-11258-bdc934" containerID="cri-o://92cc330f7714d4881734e43633b0b2097f859d6aa923699d93f81513c918bbf7" gracePeriod=2
Oct 30 04:43:55.753407 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:55.753376    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-bdc934\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="ba3cba4f-9d4c-4d0a-878e-795a7f947244" pod="opendatahub/test-curl-11258-bdc934"
Oct 30 04:43:55.939285 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:55.939251    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-0cd42e"
Oct 30 04:43:55.953838 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:55.953797    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11258-bdc934\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="ba3cba4f-9d4c-4d0a-878e-795a7f947244" pod="opendatahub/test-curl-11258-bdc934"
Oct 30 04:43:56.047749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.047674    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b242228-c33a-487f-9021-c307c5c2ce21-kube-api-access-nrjf6\" (UniqueName: \"kubernetes.io/projected/5b242228-c33a-487f-9021-c307c5c2ce21-kube-api-access-nrjf6\") pod \"5b242228-c33a-487f-9021-c307c5c2ce21\" (UID: \"5b242228-c33a-487f-9021-c307c5c2ce21\") "
Oct 30 04:43:56.049847 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.049819    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b242228-c33a-487f-9021-c307c5c2ce21-kube-api-access-nrjf6" pod "5b242228-c33a-487f-9021-c307c5c2ce21" (UID: "5b242228-c33a-487f-9021-c307c5c2ce21"). InnerVolumeSpecName "kube-api-access-nrjf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:56.149052 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.149028    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrjf6\" (UniqueName: \"kubernetes.io/projected/5b242228-c33a-487f-9021-c307c5c2ce21-kube-api-access-nrjf6\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:56.409656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.409575    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b242228-c33a-487f-9021-c307c5c2ce21" path="/var/lib/kubelet/pods/5b242228-c33a-487f-9021-c307c5c2ce21/volumes"
Oct 30 04:43:56.427611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.418456    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"]
Oct 30 04:43:56.427611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.419191    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ba3cba4f-9d4c-4d0a-878e-795a7f947244" containerName="test-curl-11258-bdc934"
Oct 30 04:43:56.427611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.419209    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3cba4f-9d4c-4d0a-878e-795a7f947244" containerName="test-curl-11258-bdc934"
Oct 30 04:43:56.427611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.419342    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3cba4f-9d4c-4d0a-878e-795a7f947244" containerName="test-curl-11258-bdc934"
Oct 30 04:43:56.427611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.425464    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.440196 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.440173    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"]
Oct 30 04:43:56.449548 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.449496    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ea1940\"/\"openshift-service-ca.crt\""
Oct 30 04:43:56.449692 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.449562    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ea1940\"/\"kube-root-ca.crt\""
Oct 30 04:43:56.454038 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.454019    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ea1940\"/\"noten-model-ea1940-kserve-self-signed-certs\""
Oct 30 04:43:56.454118 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.454056    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ea1940\"/\"default-dockercfg-fbcwz\""
Oct 30 04:43:56.554540 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.554499    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpfjf\" (UniqueName: \"kubernetes.io/projected/9c332e86-22d4-4c90-ab33-950fc2df8de9-kube-api-access-lpfjf\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.554736 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.554584    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-model-cache\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.554798 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.554727    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-dshm\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.554881 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.554862    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-home\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.554943 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.554894    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-tmp-dir\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.555000 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.554954    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c332e86-22d4-4c90-ab33-950fc2df8de9-tls-certs\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.655660 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.655630    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpfjf\" (UniqueName: \"kubernetes.io/projected/9c332e86-22d4-4c90-ab33-950fc2df8de9-kube-api-access-lpfjf\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.655834 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.655699    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-model-cache\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.655834 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.655740    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-dshm\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.655834 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.655807    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-home\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.655834 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.655830    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-tmp-dir\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.656126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.655888    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c332e86-22d4-4c90-ab33-950fc2df8de9-tls-certs\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.656212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.656190    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-model-cache\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.656287 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.656264    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-home\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.656480 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.656436    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-tmp-dir\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.658843 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.658813    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c332e86-22d4-4c90-ab33-950fc2df8de9-tls-certs\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.659091 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.659052    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-dshm\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.684628 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.684560    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpfjf\" (UniqueName: \"kubernetes.io/projected/9c332e86-22d4-4c90-ab33-950fc2df8de9-kube-api-access-lpfjf\") pod \"noten-model-ea1940-kserve-54c7669f5c-b74xz\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") " pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.747337 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.747304    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:56.758281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.758248    5072 generic.go:356] "Generic (PLEG): container finished" podID="5b242228-c33a-487f-9021-c307c5c2ce21" containerID="337cd3b2129743236705daacd15115fd078679a5667689b589ff54604d4f384d" exitCode=137
Oct 30 04:43:56.758428 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.758406    5072 scope.go:122] "RemoveContainer" containerID="337cd3b2129743236705daacd15115fd078679a5667689b589ff54604d4f384d"
Oct 30 04:43:56.758524 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.758509    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-0cd42e"
Oct 30 04:43:56.805154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.803241    5072 scope.go:122] "RemoveContainer" containerID="337cd3b2129743236705daacd15115fd078679a5667689b589ff54604d4f384d"
Oct 30 04:43:56.805154 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:56.803706    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337cd3b2129743236705daacd15115fd078679a5667689b589ff54604d4f384d\": container with ID starting with 337cd3b2129743236705daacd15115fd078679a5667689b589ff54604d4f384d not found: ID does not exist" containerID="337cd3b2129743236705daacd15115fd078679a5667689b589ff54604d4f384d"
Oct 30 04:43:56.805154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:56.803745    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337cd3b2129743236705daacd15115fd078679a5667689b589ff54604d4f384d"} err="failed to get container status \"337cd3b2129743236705daacd15115fd078679a5667689b589ff54604d4f384d\": rpc error: code = NotFound desc = could not find container \"337cd3b2129743236705daacd15115fd078679a5667689b589ff54604d4f384d\": container with ID starting with 337cd3b2129743236705daacd15115fd078679a5667689b589ff54604d4f384d not found: ID does not exist"
Oct 30 04:43:57.081991 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:57.081964    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"]
Oct 30 04:43:57.083216 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:43:57.083185    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c332e86_22d4_4c90_ab33_950fc2df8de9.slice/crio-5e875f9563dc02372129548d7ed88a829b9d940a4de5388342fcd1196425db52 WatchSource:0}: Error finding container 5e875f9563dc02372129548d7ed88a829b9d940a4de5388342fcd1196425db52: Status 404 returned error can't find the container with id 5e875f9563dc02372129548d7ed88a829b9d940a4de5388342fcd1196425db52
Oct 30 04:43:57.769057 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:57.769022    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz" event={"ID":"9c332e86-22d4-4c90-ab33-950fc2df8de9","Type":"ContainerStarted","Data":"ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4"}
Oct 30 04:43:57.769385 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:57.769145    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz" event={"ID":"9c332e86-22d4-4c90-ab33-950fc2df8de9","Type":"ContainerStarted","Data":"5e875f9563dc02372129548d7ed88a829b9d940a4de5388342fcd1196425db52"}
Oct 30 04:43:57.814394 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:57.814346    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz" podStartSLOduration=1.814331908 podStartE2EDuration="1.814331908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:57.811557552 +0000 UTC m=+2817.552980649" watchObservedRunningTime="2026-10-30 04:43:57.814331908 +0000 UTC m=+2817.555755004"
Oct 30 04:43:57.946603 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:57.946522    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-bdc934"
Oct 30 04:43:58.076565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:58.076483    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3cba4f-9d4c-4d0a-878e-795a7f947244-kube-api-access-z88md\" (UniqueName: \"kubernetes.io/projected/ba3cba4f-9d4c-4d0a-878e-795a7f947244-kube-api-access-z88md\") pod \"ba3cba4f-9d4c-4d0a-878e-795a7f947244\" (UID: \"ba3cba4f-9d4c-4d0a-878e-795a7f947244\") "
Oct 30 04:43:58.078957 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:58.078930    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3cba4f-9d4c-4d0a-878e-795a7f947244-kube-api-access-z88md" pod "ba3cba4f-9d4c-4d0a-878e-795a7f947244" (UID: "ba3cba4f-9d4c-4d0a-878e-795a7f947244"). InnerVolumeSpecName "kube-api-access-z88md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:58.178615 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:58.178581    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z88md\" (UniqueName: \"kubernetes.io/projected/ba3cba4f-9d4c-4d0a-878e-795a7f947244-kube-api-access-z88md\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:58.409457 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:58.409377    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3cba4f-9d4c-4d0a-878e-795a7f947244" path="/var/lib/kubelet/pods/ba3cba4f-9d4c-4d0a-878e-795a7f947244/volumes"
Oct 30 04:43:58.748713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:58.748687    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:43:58.776458 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:58.776420    5072 generic.go:356] "Generic (PLEG): container finished" podID="ba3cba4f-9d4c-4d0a-878e-795a7f947244" containerID="92cc330f7714d4881734e43633b0b2097f859d6aa923699d93f81513c918bbf7" exitCode=137
Oct 30 04:43:58.776872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:58.776476    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11258-bdc934"
Oct 30 04:43:58.776872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:58.776520    5072 scope.go:122] "RemoveContainer" containerID="92cc330f7714d4881734e43633b0b2097f859d6aa923699d93f81513c918bbf7"
Oct 30 04:43:58.787173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:58.787156    5072 scope.go:122] "RemoveContainer" containerID="92cc330f7714d4881734e43633b0b2097f859d6aa923699d93f81513c918bbf7"
Oct 30 04:43:58.787424 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:58.787404    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92cc330f7714d4881734e43633b0b2097f859d6aa923699d93f81513c918bbf7\": container with ID starting with 92cc330f7714d4881734e43633b0b2097f859d6aa923699d93f81513c918bbf7 not found: ID does not exist" containerID="92cc330f7714d4881734e43633b0b2097f859d6aa923699d93f81513c918bbf7"
Oct 30 04:43:58.787488 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:58.787435    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92cc330f7714d4881734e43633b0b2097f859d6aa923699d93f81513c918bbf7"} err="failed to get container status \"92cc330f7714d4881734e43633b0b2097f859d6aa923699d93f81513c918bbf7\": rpc error: code = NotFound desc = could not find container \"92cc330f7714d4881734e43633b0b2097f859d6aa923699d93f81513c918bbf7\": container with ID starting with 92cc330f7714d4881734e43633b0b2097f859d6aa923699d93f81513c918bbf7 not found: ID does not exist"
Oct 30 04:43:59.307044 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.307001    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz" podUID="b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" containerName="main" containerID="cri-o://55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491" gracePeriod=45
Oct 30 04:43:59.469092 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.469025    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz" podUID="b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": dial tcp 10.132.0.107:8000: connect: connection refused"
Oct 30 04:43:59.589083 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.589055    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:43:59.693702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.693671    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-kube-api-access-mtpxk\" (UniqueName: \"kubernetes.io/projected/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-kube-api-access-mtpxk\") pod \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") "
Oct 30 04:43:59.693702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.693705    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tls-certs\") pod \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") "
Oct 30 04:43:59.693963 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.693737    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tmp-dir\") pod \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") "
Oct 30 04:43:59.693963 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.693814    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-model-cache\") pod \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") "
Oct 30 04:43:59.693963 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.693886    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-dshm\") pod \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") "
Oct 30 04:43:59.693963 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.693910    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-home\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-home\") pod \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\" (UID: \"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b\") "
Oct 30 04:43:59.694215 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.694032    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tmp-dir" pod "b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" (UID: "b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:59.694215 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.694141    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-model-cache" pod "b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" (UID: "b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:59.694314 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.694237    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-home" pod "b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" (UID: "b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:59.694453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.694437    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:59.694500 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.694478    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:59.694500 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.694494    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:59.696162 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.696142    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tls-certs" pod "b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" (UID: "b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:43:59.696599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.696579    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-dshm" pod "b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" (UID: "b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:59.696671 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.696602    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-kube-api-access-mtpxk" pod "b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" (UID: "b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b"). InnerVolumeSpecName "kube-api-access-mtpxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:59.786171 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.786137    5072 generic.go:356] "Generic (PLEG): container finished" podID="b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" containerID="55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491" exitCode=0
Oct 30 04:43:59.786571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.786179    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"
Oct 30 04:43:59.786571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.786226    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz" event={"ID":"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b","Type":"ContainerDied","Data":"55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491"}
Oct 30 04:43:59.786571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.786271    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz" event={"ID":"b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b","Type":"ContainerDied","Data":"8ce257b4221131659f6dab37436fd8b2f7ee4cb1d302ad1591e0d79b6d6dc44f"}
Oct 30 04:43:59.786571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.786293    5072 scope.go:122] "RemoveContainer" containerID="55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491"
Oct 30 04:43:59.795316 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.795293    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtpxk\" (UniqueName: \"kubernetes.io/projected/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-kube-api-access-mtpxk\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:59.795414 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.795318    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:59.795414 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.795333    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:43:59.798405 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.798388    5072 scope.go:122] "RemoveContainer" containerID="55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491"
Oct 30 04:43:59.798687 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:43:59.798669    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491\": container with ID starting with 55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491 not found: ID does not exist" containerID="55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491"
Oct 30 04:43:59.798743 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.798698    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491"} err="failed to get container status \"55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491\": rpc error: code = NotFound desc = could not find container \"55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491\": container with ID starting with 55eee8b2c599c290eb33938a467d646b304859041bf3c20709b7ac2e435a8491 not found: ID does not exist"
Oct 30 04:43:59.842734 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.842657    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"]
Oct 30 04:43:59.850242 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:43:59.850213    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-66217b/auth-test-model-66217b-kserve-5c4f66b785-c96zz"]
Oct 30 04:44:00.409023 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:00.408995    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" path="/var/lib/kubelet/pods/b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b/volumes"
Oct 30 04:44:02.493374 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.493346    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"]
Oct 30 04:44:02.494208 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.494188    5072 cpu_manager.go:414] "containerMap: removing container" podUID="b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" containerName="main"
Oct 30 04:44:02.494297 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.494210    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" containerName="main"
Oct 30 04:44:02.494410 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.494396    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8fe6eaf-5aae-4ebe-9fe1-82ba7124039b" containerName="main"
Oct 30 04:44:02.499765 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.499744    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"
Oct 30 04:44:02.517639 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.517616    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"]
Oct 30 04:44:02.526214 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.526190    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ed8266\""
Oct 30 04:44:02.526333 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.526233    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ed8266-dockercfg-fp7vh\""
Oct 30 04:44:02.611709 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.611673    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"]
Oct 30 04:44:02.616034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.615993    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"
Oct 30 04:44:02.626842 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.625902    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fclf\" (UniqueName: \"kubernetes.io/projected/816b1bce-0e5a-4bd2-892a-6b128607ddef-kube-api-access-2fclf\") pod \"payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t\" (UID: \"816b1bce-0e5a-4bd2-892a-6b128607ddef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"
Oct 30 04:44:02.626842 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.625973    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816b1bce-0e5a-4bd2-892a-6b128607ddef-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t\" (UID: \"816b1bce-0e5a-4bd2-892a-6b128607ddef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"
Oct 30 04:44:02.645330 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.645296    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"]
Oct 30 04:44:02.727188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.727155    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9lv5\" (UniqueName: \"kubernetes.io/projected/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-kube-api-access-v9lv5\") pod \"payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55\" (UID: \"a4f79ddb-940f-4136-bc52-9c4a8e99ac64\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"
Oct 30 04:44:02.727383 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.727238    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fclf\" (UniqueName: \"kubernetes.io/projected/816b1bce-0e5a-4bd2-892a-6b128607ddef-kube-api-access-2fclf\") pod \"payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t\" (UID: \"816b1bce-0e5a-4bd2-892a-6b128607ddef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"
Oct 30 04:44:02.727383 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.727285    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816b1bce-0e5a-4bd2-892a-6b128607ddef-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t\" (UID: \"816b1bce-0e5a-4bd2-892a-6b128607ddef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"
Oct 30 04:44:02.727383 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.727302    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55\" (UID: \"a4f79ddb-940f-4136-bc52-9c4a8e99ac64\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"
Oct 30 04:44:02.728013 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.727985    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816b1bce-0e5a-4bd2-892a-6b128607ddef-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t\" (UID: \"816b1bce-0e5a-4bd2-892a-6b128607ddef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"
Oct 30 04:44:02.751688 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.751633    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fclf\" (UniqueName: \"kubernetes.io/projected/816b1bce-0e5a-4bd2-892a-6b128607ddef-kube-api-access-2fclf\") pod \"payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t\" (UID: \"816b1bce-0e5a-4bd2-892a-6b128607ddef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"
Oct 30 04:44:02.813823 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.813798    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"
Oct 30 04:44:02.828510 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.828485    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9lv5\" (UniqueName: \"kubernetes.io/projected/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-kube-api-access-v9lv5\") pod \"payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55\" (UID: \"a4f79ddb-940f-4136-bc52-9c4a8e99ac64\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"
Oct 30 04:44:02.828632 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.828554    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55\" (UID: \"a4f79ddb-940f-4136-bc52-9c4a8e99ac64\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"
Oct 30 04:44:02.829368 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.829345    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55\" (UID: \"a4f79ddb-940f-4136-bc52-9c4a8e99ac64\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"
Oct 30 04:44:02.852608 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.852580    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9lv5\" (UniqueName: \"kubernetes.io/projected/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-kube-api-access-v9lv5\") pod \"payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55\" (UID: \"a4f79ddb-940f-4136-bc52-9c4a8e99ac64\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"
Oct 30 04:44:02.955004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:02.954972    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"
Oct 30 04:44:03.121569 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:44:03.121541    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816b1bce_0e5a_4bd2_892a_6b128607ddef.slice/crio-207018e491cac2355f770ced8e6a7abd176bded20dd3678a32bfd3188282cbea WatchSource:0}: Error finding container 207018e491cac2355f770ced8e6a7abd176bded20dd3678a32bfd3188282cbea: Status 404 returned error can't find the container with id 207018e491cac2355f770ced8e6a7abd176bded20dd3678a32bfd3188282cbea
Oct 30 04:44:03.122021 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:03.121997    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"]
Oct 30 04:44:03.255920 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:03.255891    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"]
Oct 30 04:44:03.256040 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:44:03.255946    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f79ddb_940f_4136_bc52_9c4a8e99ac64.slice/crio-e3381395b8ee285e586f52e81f71938e2b6b5c0a80f293136a7432b16906ed4c WatchSource:0}: Error finding container e3381395b8ee285e586f52e81f71938e2b6b5c0a80f293136a7432b16906ed4c: Status 404 returned error can't find the container with id e3381395b8ee285e586f52e81f71938e2b6b5c0a80f293136a7432b16906ed4c
Oct 30 04:44:03.817280 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:03.817240    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t" event={"ID":"816b1bce-0e5a-4bd2-892a-6b128607ddef","Type":"ContainerStarted","Data":"369d24ddba59fbb3974e7e35b74f03c0175973d27fb915e692ac4a16b6d2d202"}
Oct 30 04:44:03.817705 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:03.817310    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"
Oct 30 04:44:03.817705 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:03.817331    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t" event={"ID":"816b1bce-0e5a-4bd2-892a-6b128607ddef","Type":"ContainerStarted","Data":"207018e491cac2355f770ced8e6a7abd176bded20dd3678a32bfd3188282cbea"}
Oct 30 04:44:03.818988 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:03.818964    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55" event={"ID":"a4f79ddb-940f-4136-bc52-9c4a8e99ac64","Type":"ContainerStarted","Data":"fe88e0cfb8f0202f3252c125d2ddf6b6c8167a923b76b7f870ddf8fa2e54a94d"}
Oct 30 04:44:03.819115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:03.819010    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55" event={"ID":"a4f79ddb-940f-4136-bc52-9c4a8e99ac64","Type":"ContainerStarted","Data":"e3381395b8ee285e586f52e81f71938e2b6b5c0a80f293136a7432b16906ed4c"}
Oct 30 04:44:03.819115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:03.819042    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"
Oct 30 04:44:03.860903 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:03.860841    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t" podStartSLOduration=1.8608243770000001 podStartE2EDuration="1.860824377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:03.859661241 +0000 UTC m=+2823.601084338" watchObservedRunningTime="2026-10-30 04:44:03.860824377 +0000 UTC m=+2823.602247477"
Oct 30 04:44:03.902518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:03.902468    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55" podStartSLOduration=1.902454224 podStartE2EDuration="1.902454224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:03.901153412 +0000 UTC m=+2823.642576522" watchObservedRunningTime="2026-10-30 04:44:03.902454224 +0000 UTC m=+2823.643877320"
Oct 30 04:44:04.962268 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:04.962237    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"]
Oct 30 04:44:11.756316 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:44:11.756238    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:44:14.828887 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:14.828857    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"
Oct 30 04:44:14.829436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:14.828925    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"
Oct 30 04:44:19.796267 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:19.796235    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"]
Oct 30 04:44:19.801736 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:19.801716    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:19.819679 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:19.819655    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"]
Oct 30 04:44:19.828458 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:19.828429    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:44:19.828648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:19.828438    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:44:19.906753 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:19.906723    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-629706a9-vmj4n\" (UID: \"9ab420b2-4283-4d6f-9271-df62eed08062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:19.906891 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:19.906789    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-629706a9-vmj4n\" (UID: \"9ab420b2-4283-4d6f-9271-df62eed08062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:19.906995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:19.906974    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxp6s\" (UniqueName: \"kubernetes.io/projected/9ab420b2-4283-4d6f-9271-df62eed08062-kube-api-access-zxp6s\") pod \"maas-api-revoke-sub-629706a9-vmj4n\" (UID: \"9ab420b2-4283-4d6f-9271-df62eed08062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:19.987406 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:19.987348    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz" podUID="9c332e86-22d4-4c90-ab33-950fc2df8de9" containerName="main" containerID="cri-o://ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4" gracePeriod=45
Oct 30 04:44:20.008511 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.008461    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-629706a9-vmj4n\" (UID: \"9ab420b2-4283-4d6f-9271-df62eed08062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:20.008659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.008640    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxp6s\" (UniqueName: \"kubernetes.io/projected/9ab420b2-4283-4d6f-9271-df62eed08062-kube-api-access-zxp6s\") pod \"maas-api-revoke-sub-629706a9-vmj4n\" (UID: \"9ab420b2-4283-4d6f-9271-df62eed08062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:20.008742 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.008682    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-629706a9-vmj4n\" (UID: \"9ab420b2-4283-4d6f-9271-df62eed08062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:20.009430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.009407    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-629706a9-vmj4n\" (UID: \"9ab420b2-4283-4d6f-9271-df62eed08062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:20.011332 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.011310    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-629706a9-vmj4n\" (UID: \"9ab420b2-4283-4d6f-9271-df62eed08062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:20.032645 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.032618    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxp6s\" (UniqueName: \"kubernetes.io/projected/9ab420b2-4283-4d6f-9271-df62eed08062-kube-api-access-zxp6s\") pod \"maas-api-revoke-sub-629706a9-vmj4n\" (UID: \"9ab420b2-4283-4d6f-9271-df62eed08062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:20.116573 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.116543    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:20.286994 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.286966    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:44:20.313664 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.313635    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-model-cache\") pod \"9c332e86-22d4-4c90-ab33-950fc2df8de9\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") "
Oct 30 04:44:20.313813 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.313691    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-dshm\") pod \"9c332e86-22d4-4c90-ab33-950fc2df8de9\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") "
Oct 30 04:44:20.313813 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.313756    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-tmp-dir\") pod \"9c332e86-22d4-4c90-ab33-950fc2df8de9\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") "
Oct 30 04:44:20.313940 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.313831    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-model-cache" pod "9c332e86-22d4-4c90-ab33-950fc2df8de9" (UID: "9c332e86-22d4-4c90-ab33-950fc2df8de9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:20.314008 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.313938    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-home\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-home\") pod \"9c332e86-22d4-4c90-ab33-950fc2df8de9\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") "
Oct 30 04:44:20.314068 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.314009    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c332e86-22d4-4c90-ab33-950fc2df8de9-kube-api-access-lpfjf\" (UniqueName: \"kubernetes.io/projected/9c332e86-22d4-4c90-ab33-950fc2df8de9-kube-api-access-lpfjf\") pod \"9c332e86-22d4-4c90-ab33-950fc2df8de9\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") "
Oct 30 04:44:20.314068 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.314048    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c332e86-22d4-4c90-ab33-950fc2df8de9-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c332e86-22d4-4c90-ab33-950fc2df8de9-tls-certs\") pod \"9c332e86-22d4-4c90-ab33-950fc2df8de9\" (UID: \"9c332e86-22d4-4c90-ab33-950fc2df8de9\") "
Oct 30 04:44:20.314068 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.314047    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-tmp-dir" pod "9c332e86-22d4-4c90-ab33-950fc2df8de9" (UID: "9c332e86-22d4-4c90-ab33-950fc2df8de9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:20.314263 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.314220    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-home" pod "9c332e86-22d4-4c90-ab33-950fc2df8de9" (UID: "9c332e86-22d4-4c90-ab33-950fc2df8de9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:20.314510 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.314492    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:20.314584 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.314515    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:20.314584 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.314528    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:20.316240 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.316218    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c332e86-22d4-4c90-ab33-950fc2df8de9-kube-api-access-lpfjf" pod "9c332e86-22d4-4c90-ab33-950fc2df8de9" (UID: "9c332e86-22d4-4c90-ab33-950fc2df8de9"). InnerVolumeSpecName "kube-api-access-lpfjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:44:20.316240 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.316231    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-dshm" pod "9c332e86-22d4-4c90-ab33-950fc2df8de9" (UID: "9c332e86-22d4-4c90-ab33-950fc2df8de9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:20.316453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.316425    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c332e86-22d4-4c90-ab33-950fc2df8de9-tls-certs" pod "9c332e86-22d4-4c90-ab33-950fc2df8de9" (UID: "9c332e86-22d4-4c90-ab33-950fc2df8de9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:44:20.331104 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.331054    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"]
Oct 30 04:44:20.331901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.331875    5072 cpu_manager.go:414] "containerMap: removing container" podUID="9c332e86-22d4-4c90-ab33-950fc2df8de9" containerName="main"
Oct 30 04:44:20.331901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.331895    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c332e86-22d4-4c90-ab33-950fc2df8de9" containerName="main"
Oct 30 04:44:20.332058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.332047    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c332e86-22d4-4c90-ab33-950fc2df8de9" containerName="main"
Oct 30 04:44:20.335707 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.335690    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:20.354303 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.354104    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"]
Oct 30 04:44:20.415470 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.415393    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggv6v\" (UniqueName: \"kubernetes.io/projected/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-kube-api-access-ggv6v\") pod \"maas-api-revoke-sub-4219c66e-l7n6z\" (UID: \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:20.415470 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.415442    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4219c66e-l7n6z\" (UID: \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:20.415696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.415527    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4219c66e-l7n6z\" (UID: \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:20.415765 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.415700    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpfjf\" (UniqueName: \"kubernetes.io/projected/9c332e86-22d4-4c90-ab33-950fc2df8de9-kube-api-access-lpfjf\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:20.415765 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.415744    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c332e86-22d4-4c90-ab33-950fc2df8de9-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:20.415765 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.415755    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c332e86-22d4-4c90-ab33-950fc2df8de9-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:20.428066 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:44:20.428046    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab420b2_4283_4d6f_9271_df62eed08062.slice/crio-00c57124d88e7990fbea17a3e98462e7a42a82df884d5d53a1d23c4586f2f433 WatchSource:0}: Error finding container 00c57124d88e7990fbea17a3e98462e7a42a82df884d5d53a1d23c4586f2f433: Status 404 returned error can't find the container with id 00c57124d88e7990fbea17a3e98462e7a42a82df884d5d53a1d23c4586f2f433
Oct 30 04:44:20.428792 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.428772    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"]
Oct 30 04:44:20.517612 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.517579    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggv6v\" (UniqueName: \"kubernetes.io/projected/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-kube-api-access-ggv6v\") pod \"maas-api-revoke-sub-4219c66e-l7n6z\" (UID: \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:20.517793 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.517648    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4219c66e-l7n6z\" (UID: \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:20.517793 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.517695    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4219c66e-l7n6z\" (UID: \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:20.518586 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.518490    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4219c66e-l7n6z\" (UID: \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:20.520515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.520495    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4219c66e-l7n6z\" (UID: \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:20.541699 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.541674    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggv6v\" (UniqueName: \"kubernetes.io/projected/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-kube-api-access-ggv6v\") pod \"maas-api-revoke-sub-4219c66e-l7n6z\" (UID: \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:20.656383 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.656352    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:20.935771 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.935736    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n" event={"ID":"9ab420b2-4283-4d6f-9271-df62eed08062","Type":"ContainerStarted","Data":"af21f07b4e8c5bebab18ddd5bfacdeadda829257d30e25ced3569d99614a5a41"}
Oct 30 04:44:20.936199 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.935796    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n" event={"ID":"9ab420b2-4283-4d6f-9271-df62eed08062","Type":"ContainerStarted","Data":"00c57124d88e7990fbea17a3e98462e7a42a82df884d5d53a1d23c4586f2f433"}
Oct 30 04:44:20.937493 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.937463    5072 generic.go:356] "Generic (PLEG): container finished" podID="9c332e86-22d4-4c90-ab33-950fc2df8de9" containerID="ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4" exitCode=0
Oct 30 04:44:20.937628 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.937514    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"
Oct 30 04:44:20.937628 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.937518    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz" event={"ID":"9c332e86-22d4-4c90-ab33-950fc2df8de9","Type":"ContainerDied","Data":"ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4"}
Oct 30 04:44:20.937628 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.937608    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz" event={"ID":"9c332e86-22d4-4c90-ab33-950fc2df8de9","Type":"ContainerDied","Data":"5e875f9563dc02372129548d7ed88a829b9d940a4de5388342fcd1196425db52"}
Oct 30 04:44:20.937749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.937631    5072 scope.go:122] "RemoveContainer" containerID="ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4"
Oct 30 04:44:20.947437 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.947410    5072 scope.go:122] "RemoveContainer" containerID="ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4"
Oct 30 04:44:20.947700 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:44:20.947679    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4\": container with ID starting with ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4 not found: ID does not exist" containerID="ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4"
Oct 30 04:44:20.947768 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.947711    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4"} err="failed to get container status \"ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4\": rpc error: code = NotFound desc = could not find container \"ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4\": container with ID starting with ca08d5dbeff4f98751fea150894e25b3c60dbb7d64b618385440508b436a4bf4 not found: ID does not exist"
Oct 30 04:44:20.958602 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:44:20.958581    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72c435c_feac_4ef5_9dce_c6f23a90e7ae.slice/crio-572d090fc7e8ffa022f03cc6d8867d8521dc1f37a27fae99d00b24dad5d2d843 WatchSource:0}: Error finding container 572d090fc7e8ffa022f03cc6d8867d8521dc1f37a27fae99d00b24dad5d2d843: Status 404 returned error can't find the container with id 572d090fc7e8ffa022f03cc6d8867d8521dc1f37a27fae99d00b24dad5d2d843
Oct 30 04:44:20.961787 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.961761    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"]
Oct 30 04:44:20.980254 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:20.980215    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n" podStartSLOduration=1.980200094 podStartE2EDuration="1.980200094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:20.976377655 +0000 UTC m=+2840.717800752" watchObservedRunningTime="2026-10-30 04:44:20.980200094 +0000 UTC m=+2840.721623192"
Oct 30 04:44:21.014570 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:21.014548    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"]
Oct 30 04:44:21.018942 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:21.018918    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ea1940/noten-model-ea1940-kserve-54c7669f5c-b74xz"]
Oct 30 04:44:21.945191 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:21.945161    5072 generic.go:356] "Generic (PLEG): container finished" podID="a72c435c-feac-4ef5-9dce-c6f23a90e7ae" containerID="6db39da4b9da8c2635831a6edcc751dc8576e8c00ecd5cbe2829f8f3520a96ae" exitCode=0
Oct 30 04:44:21.945659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:21.945247    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z" event={"ID":"a72c435c-feac-4ef5-9dce-c6f23a90e7ae","Type":"ContainerDied","Data":"6db39da4b9da8c2635831a6edcc751dc8576e8c00ecd5cbe2829f8f3520a96ae"}
Oct 30 04:44:21.945659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:21.945309    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z" event={"ID":"a72c435c-feac-4ef5-9dce-c6f23a90e7ae","Type":"ContainerStarted","Data":"572d090fc7e8ffa022f03cc6d8867d8521dc1f37a27fae99d00b24dad5d2d843"}
Oct 30 04:44:21.946957 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:21.946931    5072 generic.go:356] "Generic (PLEG): container finished" podID="9ab420b2-4283-4d6f-9271-df62eed08062" containerID="af21f07b4e8c5bebab18ddd5bfacdeadda829257d30e25ced3569d99614a5a41" exitCode=0
Oct 30 04:44:21.947200 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:21.946988    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n" event={"ID":"9ab420b2-4283-4d6f-9271-df62eed08062","Type":"ContainerDied","Data":"af21f07b4e8c5bebab18ddd5bfacdeadda829257d30e25ced3569d99614a5a41"}
Oct 30 04:44:22.408419 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:22.408388    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c332e86-22d4-4c90-ab33-950fc2df8de9" path="/var/lib/kubelet/pods/9c332e86-22d4-4c90-ab33-950fc2df8de9/volumes"
Oct 30 04:44:23.148611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.148580    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:23.172544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.172518    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:23.247228 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.247139    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-openshift-ca\") pod \"9ab420b2-4283-4d6f-9271-df62eed08062\" (UID: \"9ab420b2-4283-4d6f-9271-df62eed08062\") "
Oct 30 04:44:23.247393 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.247237    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-openshift-ca\") pod \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\" (UID: \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\") "
Oct 30 04:44:23.247393 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.247276    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-xks-ca\") pod \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\" (UID: \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\") "
Oct 30 04:44:23.247393 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.247375    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab420b2-4283-4d6f-9271-df62eed08062-kube-api-access-zxp6s\" (UniqueName: \"kubernetes.io/projected/9ab420b2-4283-4d6f-9271-df62eed08062-kube-api-access-zxp6s\") pod \"9ab420b2-4283-4d6f-9271-df62eed08062\" (UID: \"9ab420b2-4283-4d6f-9271-df62eed08062\") "
Oct 30 04:44:23.247563 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.247512    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-kube-api-access-ggv6v\" (UniqueName: \"kubernetes.io/projected/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-kube-api-access-ggv6v\") pod \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\" (UID: \"a72c435c-feac-4ef5-9dce-c6f23a90e7ae\") "
Oct 30 04:44:23.247563 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.247537    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-openshift-ca" pod "9ab420b2-4283-4d6f-9271-df62eed08062" (UID: "9ab420b2-4283-4d6f-9271-df62eed08062"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:44:23.247563 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.247558    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-xks-ca\") pod \"9ab420b2-4283-4d6f-9271-df62eed08062\" (UID: \"9ab420b2-4283-4d6f-9271-df62eed08062\") "
Oct 30 04:44:23.247721 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.247614    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-openshift-ca" pod "a72c435c-feac-4ef5-9dce-c6f23a90e7ae" (UID: "a72c435c-feac-4ef5-9dce-c6f23a90e7ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:44:23.247996 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.247970    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:23.248142 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.248000    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:23.250576 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.250539    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab420b2-4283-4d6f-9271-df62eed08062-kube-api-access-zxp6s" pod "9ab420b2-4283-4d6f-9271-df62eed08062" (UID: "9ab420b2-4283-4d6f-9271-df62eed08062"). InnerVolumeSpecName "kube-api-access-zxp6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:44:23.250661 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.250636    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-xks-ca" pod "a72c435c-feac-4ef5-9dce-c6f23a90e7ae" (UID: "a72c435c-feac-4ef5-9dce-c6f23a90e7ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:44:23.250861 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.250840    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-kube-api-access-ggv6v" pod "a72c435c-feac-4ef5-9dce-c6f23a90e7ae" (UID: "a72c435c-feac-4ef5-9dce-c6f23a90e7ae"). InnerVolumeSpecName "kube-api-access-ggv6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:44:23.258842 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.258819    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-xks-ca" pod "9ab420b2-4283-4d6f-9271-df62eed08062" (UID: "9ab420b2-4283-4d6f-9271-df62eed08062"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:44:23.348880 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.348843    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:23.349042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.348937    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxp6s\" (UniqueName: \"kubernetes.io/projected/9ab420b2-4283-4d6f-9271-df62eed08062-kube-api-access-zxp6s\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:23.349042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.348955    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggv6v\" (UniqueName: \"kubernetes.io/projected/a72c435c-feac-4ef5-9dce-c6f23a90e7ae-kube-api-access-ggv6v\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:23.349042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.348972    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ab420b2-4283-4d6f-9271-df62eed08062-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:23.961636 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.961608    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"
Oct 30 04:44:23.961822 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.961605    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z" event={"ID":"a72c435c-feac-4ef5-9dce-c6f23a90e7ae","Type":"ContainerDied","Data":"572d090fc7e8ffa022f03cc6d8867d8521dc1f37a27fae99d00b24dad5d2d843"}
Oct 30 04:44:23.961822 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.961744    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="572d090fc7e8ffa022f03cc6d8867d8521dc1f37a27fae99d00b24dad5d2d843"
Oct 30 04:44:23.963095 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.963056    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n" event={"ID":"9ab420b2-4283-4d6f-9271-df62eed08062","Type":"ContainerDied","Data":"00c57124d88e7990fbea17a3e98462e7a42a82df884d5d53a1d23c4586f2f433"}
Oct 30 04:44:23.963239 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.963109    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"
Oct 30 04:44:23.963239 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:23.963134    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00c57124d88e7990fbea17a3e98462e7a42a82df884d5d53a1d23c4586f2f433"
Oct 30 04:44:30.132264 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.132230    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"]
Oct 30 04:44:30.132850 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.132832    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a72c435c-feac-4ef5-9dce-c6f23a90e7ae" containerName="revoke-keys"
Oct 30 04:44:30.132850 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.132850    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72c435c-feac-4ef5-9dce-c6f23a90e7ae" containerName="revoke-keys"
Oct 30 04:44:30.132958 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.132873    5072 cpu_manager.go:414] "containerMap: removing container" podUID="9ab420b2-4283-4d6f-9271-df62eed08062" containerName="revoke-keys"
Oct 30 04:44:30.132958 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.132879    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab420b2-4283-4d6f-9271-df62eed08062" containerName="revoke-keys"
Oct 30 04:44:30.133048 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.133014    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab420b2-4283-4d6f-9271-df62eed08062" containerName="revoke-keys"
Oct 30 04:44:30.133048 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.133032    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72c435c-feac-4ef5-9dce-c6f23a90e7ae" containerName="revoke-keys"
Oct 30 04:44:30.136605 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.136588    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:30.157773 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.157746    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"]
Oct 30 04:44:30.164354 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.164329    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:44:30.164633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.164609    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:44:30.227297 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.227268    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7f76dd-wpc72\" (UID: \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:30.227412 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.227320    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcsnp\" (UniqueName: \"kubernetes.io/projected/82602269-b23c-4bbc-a5ec-c3d32d148f1b-kube-api-access-fcsnp\") pod \"maas-api-revoke-sub-fd7f76dd-wpc72\" (UID: \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:30.227497 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.227479    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7f76dd-wpc72\" (UID: \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:30.328921 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.328893    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcsnp\" (UniqueName: \"kubernetes.io/projected/82602269-b23c-4bbc-a5ec-c3d32d148f1b-kube-api-access-fcsnp\") pod \"maas-api-revoke-sub-fd7f76dd-wpc72\" (UID: \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:30.329052 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.329031    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7f76dd-wpc72\" (UID: \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:30.329151 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.329136    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7f76dd-wpc72\" (UID: \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:30.329762 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.329736    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7f76dd-wpc72\" (UID: \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:30.331515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.331496    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7f76dd-wpc72\" (UID: \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:30.353001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.352976    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcsnp\" (UniqueName: \"kubernetes.io/projected/82602269-b23c-4bbc-a5ec-c3d32d148f1b-kube-api-access-fcsnp\") pod \"maas-api-revoke-sub-fd7f76dd-wpc72\" (UID: \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:30.465162 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.465139    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:30.730418 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.730346    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"]
Oct 30 04:44:30.734769 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.734746    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:30.755497 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.755468    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"]
Oct 30 04:44:30.768524 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.768499    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"]
Oct 30 04:44:30.769595 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:44:30.769574    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82602269_b23c_4bbc_a5ec_c3d32d148f1b.slice/crio-37924d7894f6dbdc4f4d4f2af5f5b729d5b73ff4744d154ec17eb64a9f94ae37 WatchSource:0}: Error finding container 37924d7894f6dbdc4f4d4f2af5f5b729d5b73ff4744d154ec17eb64a9f94ae37: Status 404 returned error can't find the container with id 37924d7894f6dbdc4f4d4f2af5f5b729d5b73ff4744d154ec17eb64a9f94ae37
Oct 30 04:44:30.836485 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.836457    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ebd708-n5h2r\" (UID: \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:30.836610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.836597    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnpk4\" (UniqueName: \"kubernetes.io/projected/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-kube-api-access-pnpk4\") pod \"maas-api-revoke-sub-41ebd708-n5h2r\" (UID: \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:30.836673 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.836661    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ebd708-n5h2r\" (UID: \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:30.937615 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.937584    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnpk4\" (UniqueName: \"kubernetes.io/projected/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-kube-api-access-pnpk4\") pod \"maas-api-revoke-sub-41ebd708-n5h2r\" (UID: \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:30.937750 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.937664    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ebd708-n5h2r\" (UID: \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:30.937750 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.937739    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ebd708-n5h2r\" (UID: \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:30.938475 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.938453    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ebd708-n5h2r\" (UID: \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:30.940063 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.940042    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ebd708-n5h2r\" (UID: \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:30.961569 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:30.961550    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnpk4\" (UniqueName: \"kubernetes.io/projected/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-kube-api-access-pnpk4\") pod \"maas-api-revoke-sub-41ebd708-n5h2r\" (UID: \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:31.007187 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:31.007119    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72" event={"ID":"82602269-b23c-4bbc-a5ec-c3d32d148f1b","Type":"ContainerStarted","Data":"37924d7894f6dbdc4f4d4f2af5f5b729d5b73ff4744d154ec17eb64a9f94ae37"}
Oct 30 04:44:31.050264 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:31.050239    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:31.353242 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:44:31.353199    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a718c06_ccfe_4fcb_913f_29f6b21d6aa0.slice/crio-506acab9a9dc4aeaa1df795675c4dcbc4d82fee5fef10b9348bc1dcedd164e2e WatchSource:0}: Error finding container 506acab9a9dc4aeaa1df795675c4dcbc4d82fee5fef10b9348bc1dcedd164e2e: Status 404 returned error can't find the container with id 506acab9a9dc4aeaa1df795675c4dcbc4d82fee5fef10b9348bc1dcedd164e2e
Oct 30 04:44:31.353588 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:31.353298    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"]
Oct 30 04:44:32.015362 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:32.015315    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r" event={"ID":"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0","Type":"ContainerStarted","Data":"607bc75ca21c5fede1e648218a825425cbe68231be559b7b2b9c1100c1aa2a23"}
Oct 30 04:44:32.015572 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:32.015383    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r" event={"ID":"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0","Type":"ContainerStarted","Data":"506acab9a9dc4aeaa1df795675c4dcbc4d82fee5fef10b9348bc1dcedd164e2e"}
Oct 30 04:44:32.016983 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:32.016953    5072 generic.go:356] "Generic (PLEG): container finished" podID="82602269-b23c-4bbc-a5ec-c3d32d148f1b" containerID="e30a5aa74f30880cdda10a332a5d520010e69eadd366f6d4792c850984b6235f" exitCode=0
Oct 30 04:44:32.017129 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:32.017038    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72" event={"ID":"82602269-b23c-4bbc-a5ec-c3d32d148f1b","Type":"ContainerDied","Data":"e30a5aa74f30880cdda10a332a5d520010e69eadd366f6d4792c850984b6235f"}
Oct 30 04:44:32.058189 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:32.058150    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r" podStartSLOduration=2.058137866 podStartE2EDuration="2.058137866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:32.056582099 +0000 UTC m=+2851.798005197" watchObservedRunningTime="2026-10-30 04:44:32.058137866 +0000 UTC m=+2851.799560963"
Oct 30 04:44:33.031606 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:33.031569    5072 generic.go:356] "Generic (PLEG): container finished" podID="7a718c06-ccfe-4fcb-913f-29f6b21d6aa0" containerID="607bc75ca21c5fede1e648218a825425cbe68231be559b7b2b9c1100c1aa2a23" exitCode=0
Oct 30 04:44:33.032143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:33.032092    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r" event={"ID":"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0","Type":"ContainerDied","Data":"607bc75ca21c5fede1e648218a825425cbe68231be559b7b2b9c1100c1aa2a23"}
Oct 30 04:44:33.195654 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:33.195629    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:33.264050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:33.264026    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82602269-b23c-4bbc-a5ec-c3d32d148f1b-kube-api-access-fcsnp\" (UniqueName: \"kubernetes.io/projected/82602269-b23c-4bbc-a5ec-c3d32d148f1b-kube-api-access-fcsnp\") pod \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\" (UID: \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\") "
Oct 30 04:44:33.264202 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:33.264113    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-xks-ca\") pod \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\" (UID: \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\") "
Oct 30 04:44:33.264276 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:33.264263    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-openshift-ca\") pod \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\" (UID: \"82602269-b23c-4bbc-a5ec-c3d32d148f1b\") "
Oct 30 04:44:33.264670 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:33.264645    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-openshift-ca" pod "82602269-b23c-4bbc-a5ec-c3d32d148f1b" (UID: "82602269-b23c-4bbc-a5ec-c3d32d148f1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:44:33.266352 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:33.266322    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82602269-b23c-4bbc-a5ec-c3d32d148f1b-kube-api-access-fcsnp" pod "82602269-b23c-4bbc-a5ec-c3d32d148f1b" (UID: "82602269-b23c-4bbc-a5ec-c3d32d148f1b"). InnerVolumeSpecName "kube-api-access-fcsnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:44:33.266475 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:33.266435    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-xks-ca" pod "82602269-b23c-4bbc-a5ec-c3d32d148f1b" (UID: "82602269-b23c-4bbc-a5ec-c3d32d148f1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:44:33.365321 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:33.365255    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcsnp\" (UniqueName: \"kubernetes.io/projected/82602269-b23c-4bbc-a5ec-c3d32d148f1b-kube-api-access-fcsnp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:33.365321 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:33.365280    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:33.365321 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:33.365289    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82602269-b23c-4bbc-a5ec-c3d32d148f1b-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:34.039859 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.039832    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"
Oct 30 04:44:34.040304 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.039831    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72" event={"ID":"82602269-b23c-4bbc-a5ec-c3d32d148f1b","Type":"ContainerDied","Data":"37924d7894f6dbdc4f4d4f2af5f5b729d5b73ff4744d154ec17eb64a9f94ae37"}
Oct 30 04:44:34.040304 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.039971    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37924d7894f6dbdc4f4d4f2af5f5b729d5b73ff4744d154ec17eb64a9f94ae37"
Oct 30 04:44:34.205475 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.205454    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:34.275253 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.275219    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-xks-ca\") pod \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\" (UID: \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\") "
Oct 30 04:44:34.275449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.275380    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-kube-api-access-pnpk4\" (UniqueName: \"kubernetes.io/projected/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-kube-api-access-pnpk4\") pod \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\" (UID: \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\") "
Oct 30 04:44:34.275449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.275412    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-openshift-ca\") pod \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\" (UID: \"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0\") "
Oct 30 04:44:34.275818 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.275793    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-openshift-ca" pod "7a718c06-ccfe-4fcb-913f-29f6b21d6aa0" (UID: "7a718c06-ccfe-4fcb-913f-29f6b21d6aa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:44:34.277542 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.277510    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-xks-ca" pod "7a718c06-ccfe-4fcb-913f-29f6b21d6aa0" (UID: "7a718c06-ccfe-4fcb-913f-29f6b21d6aa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:44:34.277712 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.277680    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-kube-api-access-pnpk4" pod "7a718c06-ccfe-4fcb-913f-29f6b21d6aa0" (UID: "7a718c06-ccfe-4fcb-913f-29f6b21d6aa0"). InnerVolumeSpecName "kube-api-access-pnpk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:44:34.377329 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.377263    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnpk4\" (UniqueName: \"kubernetes.io/projected/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-kube-api-access-pnpk4\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:34.377329 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.377289    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:34.377329 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:34.377298    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:35.048594 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:35.048566    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"
Oct 30 04:44:35.048594 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:35.048579    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r" event={"ID":"7a718c06-ccfe-4fcb-913f-29f6b21d6aa0","Type":"ContainerDied","Data":"506acab9a9dc4aeaa1df795675c4dcbc4d82fee5fef10b9348bc1dcedd164e2e"}
Oct 30 04:44:35.049095 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:35.048628    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="506acab9a9dc4aeaa1df795675c4dcbc4d82fee5fef10b9348bc1dcedd164e2e"
Oct 30 04:44:41.751813 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:44:41.751780    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:44:42.377019 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:42.376982    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"]
Oct 30 04:44:48.262135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.262105    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11255-e26789"]
Oct 30 04:44:48.262678 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.262663    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7a718c06-ccfe-4fcb-913f-29f6b21d6aa0" containerName="revoke-keys"
Oct 30 04:44:48.262678 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.262679    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a718c06-ccfe-4fcb-913f-29f6b21d6aa0" containerName="revoke-keys"
Oct 30 04:44:48.262782 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.262709    5072 cpu_manager.go:414] "containerMap: removing container" podUID="82602269-b23c-4bbc-a5ec-c3d32d148f1b" containerName="revoke-keys"
Oct 30 04:44:48.262782 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.262715    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82602269-b23c-4bbc-a5ec-c3d32d148f1b" containerName="revoke-keys"
Oct 30 04:44:48.262848 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.262816    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a718c06-ccfe-4fcb-913f-29f6b21d6aa0" containerName="revoke-keys"
Oct 30 04:44:48.262848 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.262829    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="82602269-b23c-4bbc-a5ec-c3d32d148f1b" containerName="revoke-keys"
Oct 30 04:44:48.267440 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.267422    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11255-e26789"
Oct 30 04:44:48.282858 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.282836    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11255-e26789"]
Oct 30 04:44:48.291211 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.291185    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gsgvm\""
Oct 30 04:44:48.332675 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.332645    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjkkx\" (UniqueName: \"kubernetes.io/projected/2162c728-6ee5-4056-9ba8-c1937d10afd3-kube-api-access-fjkkx\") pod \"test-curl-11255-e26789\" (UID: \"2162c728-6ee5-4056-9ba8-c1937d10afd3\") " pod="opendatahub/test-curl-11255-e26789"
Oct 30 04:44:48.433683 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.433650    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkkx\" (UniqueName: \"kubernetes.io/projected/2162c728-6ee5-4056-9ba8-c1937d10afd3-kube-api-access-fjkkx\") pod \"test-curl-11255-e26789\" (UID: \"2162c728-6ee5-4056-9ba8-c1937d10afd3\") " pod="opendatahub/test-curl-11255-e26789"
Oct 30 04:44:48.457546 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.457516    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkkx\" (UniqueName: \"kubernetes.io/projected/2162c728-6ee5-4056-9ba8-c1937d10afd3-kube-api-access-fjkkx\") pod \"test-curl-11255-e26789\" (UID: \"2162c728-6ee5-4056-9ba8-c1937d10afd3\") " pod="opendatahub/test-curl-11255-e26789"
Oct 30 04:44:48.579856 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.579784    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11255-e26789"
Oct 30 04:44:48.881309 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:44:48.881274    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2162c728_6ee5_4056_9ba8_c1937d10afd3.slice/crio-9d75bf8e87f501031e643764a4e5a839e5be3c22072135758003e4c0f1bc240b WatchSource:0}: Error finding container 9d75bf8e87f501031e643764a4e5a839e5be3c22072135758003e4c0f1bc240b: Status 404 returned error can't find the container with id 9d75bf8e87f501031e643764a4e5a839e5be3c22072135758003e4c0f1bc240b
Oct 30 04:44:48.881521 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:48.881498    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11255-e26789"]
Oct 30 04:44:49.139729 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:49.139648    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11255-e26789" event={"ID":"2162c728-6ee5-4056-9ba8-c1937d10afd3","Type":"ContainerStarted","Data":"e9b7c8f057d0d0a891c83584b9447e0c54eb46df4e6d58315c18c1bacfc1bf1f"}
Oct 30 04:44:49.139729 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:49.139693    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11255-e26789" event={"ID":"2162c728-6ee5-4056-9ba8-c1937d10afd3","Type":"ContainerStarted","Data":"9d75bf8e87f501031e643764a4e5a839e5be3c22072135758003e4c0f1bc240b"}
Oct 30 04:44:49.180170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:49.180109    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11255-e26789" podStartSLOduration=1.180060029 podStartE2EDuration="1.180060029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:49.178154613 +0000 UTC m=+2868.919577709" watchObservedRunningTime="2026-10-30 04:44:49.180060029 +0000 UTC m=+2868.921483125"
Oct 30 04:44:50.041378 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:50.041337    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11255-e26789"]
Oct 30 04:44:50.045750 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:50.045720    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11255-e26789"]
Oct 30 04:44:50.312177 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:50.312068    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"]
Oct 30 04:44:51.152267 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:51.152199    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11255-e26789" podUID="2162c728-6ee5-4056-9ba8-c1937d10afd3" containerName="test-curl-11255-e26789" containerID="cri-o://e9b7c8f057d0d0a891c83584b9447e0c54eb46df4e6d58315c18c1bacfc1bf1f" gracePeriod=2
Oct 30 04:44:53.382159 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:53.381156    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11255-e26789"
Oct 30 04:44:53.494263 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:53.494227    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2162c728-6ee5-4056-9ba8-c1937d10afd3-kube-api-access-fjkkx\" (UniqueName: \"kubernetes.io/projected/2162c728-6ee5-4056-9ba8-c1937d10afd3-kube-api-access-fjkkx\") pod \"2162c728-6ee5-4056-9ba8-c1937d10afd3\" (UID: \"2162c728-6ee5-4056-9ba8-c1937d10afd3\") "
Oct 30 04:44:53.496586 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:53.496555    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2162c728-6ee5-4056-9ba8-c1937d10afd3-kube-api-access-fjkkx" pod "2162c728-6ee5-4056-9ba8-c1937d10afd3" (UID: "2162c728-6ee5-4056-9ba8-c1937d10afd3"). InnerVolumeSpecName "kube-api-access-fjkkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:44:53.596234 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:53.596194    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjkkx\" (UniqueName: \"kubernetes.io/projected/2162c728-6ee5-4056-9ba8-c1937d10afd3-kube-api-access-fjkkx\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:54.174835 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:54.174799    5072 generic.go:356] "Generic (PLEG): container finished" podID="2162c728-6ee5-4056-9ba8-c1937d10afd3" containerID="e9b7c8f057d0d0a891c83584b9447e0c54eb46df4e6d58315c18c1bacfc1bf1f" exitCode=137
Oct 30 04:44:54.175064 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:54.174865    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11255-e26789"
Oct 30 04:44:54.175064 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:54.174893    5072 scope.go:122] "RemoveContainer" containerID="e9b7c8f057d0d0a891c83584b9447e0c54eb46df4e6d58315c18c1bacfc1bf1f"
Oct 30 04:44:54.186766 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:54.186742    5072 scope.go:122] "RemoveContainer" containerID="e9b7c8f057d0d0a891c83584b9447e0c54eb46df4e6d58315c18c1bacfc1bf1f"
Oct 30 04:44:54.187037 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:44:54.187015    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b7c8f057d0d0a891c83584b9447e0c54eb46df4e6d58315c18c1bacfc1bf1f\": container with ID starting with e9b7c8f057d0d0a891c83584b9447e0c54eb46df4e6d58315c18c1bacfc1bf1f not found: ID does not exist" containerID="e9b7c8f057d0d0a891c83584b9447e0c54eb46df4e6d58315c18c1bacfc1bf1f"
Oct 30 04:44:54.187128 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:54.187050    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b7c8f057d0d0a891c83584b9447e0c54eb46df4e6d58315c18c1bacfc1bf1f"} err="failed to get container status \"e9b7c8f057d0d0a891c83584b9447e0c54eb46df4e6d58315c18c1bacfc1bf1f\": rpc error: code = NotFound desc = could not find container \"e9b7c8f057d0d0a891c83584b9447e0c54eb46df4e6d58315c18c1bacfc1bf1f\": container with ID starting with e9b7c8f057d0d0a891c83584b9447e0c54eb46df4e6d58315c18c1bacfc1bf1f not found: ID does not exist"
Oct 30 04:44:54.197531 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:44:54.197502    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11255-e26789\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="2162c728-6ee5-4056-9ba8-c1937d10afd3" pod="opendatahub/test-curl-11255-e26789"
Oct 30 04:44:54.408237 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:54.408204    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2162c728-6ee5-4056-9ba8-c1937d10afd3" path="/var/lib/kubelet/pods/2162c728-6ee5-4056-9ba8-c1937d10afd3/volumes"
Oct 30 04:44:57.405787 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.405736    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h" podUID="d34726f4-eae7-4909-b6d0-a3b341f4f13d" containerName="main" containerID="cri-o://b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd" gracePeriod=45
Oct 30 04:44:57.497168 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.497136    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"]
Oct 30 04:44:57.497789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.497770    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2162c728-6ee5-4056-9ba8-c1937d10afd3" containerName="test-curl-11255-e26789"
Oct 30 04:44:57.497883 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.497792    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2162c728-6ee5-4056-9ba8-c1937d10afd3" containerName="test-curl-11255-e26789"
Oct 30 04:44:57.497999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.497985    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2162c728-6ee5-4056-9ba8-c1937d10afd3" containerName="test-curl-11255-e26789"
Oct 30 04:44:57.501601 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.501579    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"
Oct 30 04:44:57.521766 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.521732    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"]
Oct 30 04:44:57.530175 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.530037    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-6378cc-dockercfg-mz45t\""
Oct 30 04:44:57.530823 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.530037    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-6378cc\""
Oct 30 04:44:57.546665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.546634    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3cb2b9f-1498-461d-bda4-4ddee9819453-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr\" (UID: \"c3cb2b9f-1498-461d-bda4-4ddee9819453\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"
Oct 30 04:44:57.546854 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.546786    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7c49\" (UniqueName: \"kubernetes.io/projected/c3cb2b9f-1498-461d-bda4-4ddee9819453-kube-api-access-m7c49\") pod \"payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr\" (UID: \"c3cb2b9f-1498-461d-bda4-4ddee9819453\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"
Oct 30 04:44:57.579205 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.579167    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"]
Oct 30 04:44:57.584561 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.584537    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"
Oct 30 04:44:57.617804 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.617551    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"]
Oct 30 04:44:57.651097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.648645    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h\" (UID: \"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"
Oct 30 04:44:57.651097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.648734    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3cb2b9f-1498-461d-bda4-4ddee9819453-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr\" (UID: \"c3cb2b9f-1498-461d-bda4-4ddee9819453\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"
Oct 30 04:44:57.651097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.648802    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f4gb\" (UniqueName: \"kubernetes.io/projected/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-kube-api-access-2f4gb\") pod \"payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h\" (UID: \"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"
Oct 30 04:44:57.651097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.649127    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7c49\" (UniqueName: \"kubernetes.io/projected/c3cb2b9f-1498-461d-bda4-4ddee9819453-kube-api-access-m7c49\") pod \"payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr\" (UID: \"c3cb2b9f-1498-461d-bda4-4ddee9819453\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"
Oct 30 04:44:57.651097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.650922    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3cb2b9f-1498-461d-bda4-4ddee9819453-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr\" (UID: \"c3cb2b9f-1498-461d-bda4-4ddee9819453\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"
Oct 30 04:44:57.680221 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.677168    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7c49\" (UniqueName: \"kubernetes.io/projected/c3cb2b9f-1498-461d-bda4-4ddee9819453-kube-api-access-m7c49\") pod \"payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr\" (UID: \"c3cb2b9f-1498-461d-bda4-4ddee9819453\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"
Oct 30 04:44:57.749935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.749907    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h\" (UID: \"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"
Oct 30 04:44:57.750115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.749987    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f4gb\" (UniqueName: \"kubernetes.io/projected/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-kube-api-access-2f4gb\") pod \"payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h\" (UID: \"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"
Oct 30 04:44:57.750810 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.750787    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h\" (UID: \"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"
Oct 30 04:44:57.759503 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.759473    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:44:57.774590 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.774563    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f4gb\" (UniqueName: \"kubernetes.io/projected/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-kube-api-access-2f4gb\") pod \"payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h\" (UID: \"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"
Oct 30 04:44:57.819979 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.819930    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"
Oct 30 04:44:57.851365 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.851333    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-home\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-home\") pod \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") "
Oct 30 04:44:57.851541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.851400    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tmp-dir\") pod \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") "
Oct 30 04:44:57.851541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.851496    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-model-cache\") pod \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") "
Oct 30 04:44:57.851541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.851528    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34726f4-eae7-4909-b6d0-a3b341f4f13d-kube-api-access-ts6vr\" (UniqueName: \"kubernetes.io/projected/d34726f4-eae7-4909-b6d0-a3b341f4f13d-kube-api-access-ts6vr\") pod \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") "
Oct 30 04:44:57.851724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.851554    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-dshm\") pod \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") "
Oct 30 04:44:57.851724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.851660    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-home" pod "d34726f4-eae7-4909-b6d0-a3b341f4f13d" (UID: "d34726f4-eae7-4909-b6d0-a3b341f4f13d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:57.851832 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.851725    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tls-certs\") pod \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\" (UID: \"d34726f4-eae7-4909-b6d0-a3b341f4f13d\") "
Oct 30 04:44:57.851832 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.851761    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tmp-dir" pod "d34726f4-eae7-4909-b6d0-a3b341f4f13d" (UID: "d34726f4-eae7-4909-b6d0-a3b341f4f13d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:57.851947 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.851881    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-model-cache" pod "d34726f4-eae7-4909-b6d0-a3b341f4f13d" (UID: "d34726f4-eae7-4909-b6d0-a3b341f4f13d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:57.852964 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.852933    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:57.852964 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.852964    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:57.853281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.852981    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:57.854426 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.854396    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-dshm" pod "d34726f4-eae7-4909-b6d0-a3b341f4f13d" (UID: "d34726f4-eae7-4909-b6d0-a3b341f4f13d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:57.854426 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.854396    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34726f4-eae7-4909-b6d0-a3b341f4f13d-kube-api-access-ts6vr" pod "d34726f4-eae7-4909-b6d0-a3b341f4f13d" (UID: "d34726f4-eae7-4909-b6d0-a3b341f4f13d"). InnerVolumeSpecName "kube-api-access-ts6vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:44:57.854587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.854496    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tls-certs" pod "d34726f4-eae7-4909-b6d0-a3b341f4f13d" (UID: "d34726f4-eae7-4909-b6d0-a3b341f4f13d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:44:57.950457 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.950421    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"
Oct 30 04:44:57.954250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.954218    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34726f4-eae7-4909-b6d0-a3b341f4f13d-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:57.954421 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.954405    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts6vr\" (UniqueName: \"kubernetes.io/projected/d34726f4-eae7-4909-b6d0-a3b341f4f13d-kube-api-access-ts6vr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:57.954496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:57.954424    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d34726f4-eae7-4909-b6d0-a3b341f4f13d-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:44:58.130878 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:44:58.130850    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cb2b9f_1498_461d_bda4_4ddee9819453.slice/crio-832fa879dfb6cfa2ad33e176cb6b623620dfa03757d611be6f0d3b4474c0b59d WatchSource:0}: Error finding container 832fa879dfb6cfa2ad33e176cb6b623620dfa03757d611be6f0d3b4474c0b59d: Status 404 returned error can't find the container with id 832fa879dfb6cfa2ad33e176cb6b623620dfa03757d611be6f0d3b4474c0b59d
Oct 30 04:44:58.131111 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.131088    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"]
Oct 30 04:44:58.210188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.210157    5072 generic.go:356] "Generic (PLEG): container finished" podID="d34726f4-eae7-4909-b6d0-a3b341f4f13d" containerID="b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd" exitCode=0
Oct 30 04:44:58.210302 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.210205    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h" event={"ID":"d34726f4-eae7-4909-b6d0-a3b341f4f13d","Type":"ContainerDied","Data":"b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd"}
Oct 30 04:44:58.210302 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.210256    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h" event={"ID":"d34726f4-eae7-4909-b6d0-a3b341f4f13d","Type":"ContainerDied","Data":"55892da10f655e52db85bbcf72a8ad26f3b125220797d560b2ab585731e8aef2"}
Oct 30 04:44:58.210302 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.210280    5072 scope.go:122] "RemoveContainer" containerID="b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd"
Oct 30 04:44:58.210630 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.210567    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"
Oct 30 04:44:58.212350 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.212324    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr" event={"ID":"c3cb2b9f-1498-461d-bda4-4ddee9819453","Type":"ContainerStarted","Data":"832fa879dfb6cfa2ad33e176cb6b623620dfa03757d611be6f0d3b4474c0b59d"}
Oct 30 04:44:58.224872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.224757    5072 scope.go:122] "RemoveContainer" containerID="b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd"
Oct 30 04:44:58.225566 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:44:58.225532    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd\": container with ID starting with b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd not found: ID does not exist" containerID="b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd"
Oct 30 04:44:58.225692 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.225578    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd"} err="failed to get container status \"b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd\": rpc error: code = NotFound desc = could not find container \"b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd\": container with ID starting with b5f829e51093255c3eb53fcab7a2a5d415392d4d551e4e45e59760ddc8dd92cd not found: ID does not exist"
Oct 30 04:44:58.263260 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:44:58.263221    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae8ce0e_895c_4c4f_826d_ddffa5a16e4d.slice/crio-90d6789d28d3afeec63a1a558631d6f168042586b9718f8fe94b2b2558fd9124 WatchSource:0}: Error finding container 90d6789d28d3afeec63a1a558631d6f168042586b9718f8fe94b2b2558fd9124: Status 404 returned error can't find the container with id 90d6789d28d3afeec63a1a558631d6f168042586b9718f8fe94b2b2558fd9124
Oct 30 04:44:58.264113 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.264063    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"]
Oct 30 04:44:58.268403 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.268378    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"]
Oct 30 04:44:58.275293 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.275263    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h"]
Oct 30 04:44:58.410707 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.410672    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d34726f4-eae7-4909-b6d0-a3b341f4f13d" path="/var/lib/kubelet/pods/d34726f4-eae7-4909-b6d0-a3b341f4f13d/volumes"
Oct 30 04:44:58.740141 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:58.740060    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-e6b837/auth-test-model-e6b837-kserve-776d7f4bd5-dlb2h" podUID="d34726f4-eae7-4909-b6d0-a3b341f4f13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Oct 30 04:44:59.110640 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.110565    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"]
Oct 30 04:44:59.111321 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.111304    5072 cpu_manager.go:414] "containerMap: removing container" podUID="d34726f4-eae7-4909-b6d0-a3b341f4f13d" containerName="main"
Oct 30 04:44:59.111379 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.111324    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34726f4-eae7-4909-b6d0-a3b341f4f13d" containerName="main"
Oct 30 04:44:59.111472 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.111463    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34726f4-eae7-4909-b6d0-a3b341f4f13d" containerName="main"
Oct 30 04:44:59.115392 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.115369    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"
Oct 30 04:44:59.131089 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.131058    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-483bea-dockercfg-qvljt\""
Oct 30 04:44:59.133795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.133771    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"]
Oct 30 04:44:59.134310 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.134257    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-483bea\""
Oct 30 04:44:59.169460 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.169424    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514edbce-f4ec-41b9-9b7a-e6879381720d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh\" (UID: \"514edbce-f4ec-41b9-9b7a-e6879381720d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"
Oct 30 04:44:59.169644 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.169569    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwgbh\" (UniqueName: \"kubernetes.io/projected/514edbce-f4ec-41b9-9b7a-e6879381720d-kube-api-access-hwgbh\") pod \"payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh\" (UID: \"514edbce-f4ec-41b9-9b7a-e6879381720d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"
Oct 30 04:44:59.206430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.206394    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"]
Oct 30 04:44:59.210941 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.210911    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"
Oct 30 04:44:59.231530 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.231500    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"]
Oct 30 04:44:59.231800 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.231756    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h" event={"ID":"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d","Type":"ContainerStarted","Data":"ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36"}
Oct 30 04:44:59.231914 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.231824    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h" event={"ID":"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d","Type":"ContainerStarted","Data":"90d6789d28d3afeec63a1a558631d6f168042586b9718f8fe94b2b2558fd9124"}
Oct 30 04:44:59.232025 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.232010    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"
Oct 30 04:44:59.236377 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.236346    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr" event={"ID":"c3cb2b9f-1498-461d-bda4-4ddee9819453","Type":"ContainerStarted","Data":"88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06"}
Oct 30 04:44:59.236520 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.236497    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"
Oct 30 04:44:59.270941 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.270795    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lnwc\" (UniqueName: \"kubernetes.io/projected/5822e703-13ed-4d49-8a12-2f51750ccbe4-kube-api-access-6lnwc\") pod \"payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q\" (UID: \"5822e703-13ed-4d49-8a12-2f51750ccbe4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"
Oct 30 04:44:59.270941 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.270941    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514edbce-f4ec-41b9-9b7a-e6879381720d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh\" (UID: \"514edbce-f4ec-41b9-9b7a-e6879381720d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"
Oct 30 04:44:59.271233 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.270988    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5822e703-13ed-4d49-8a12-2f51750ccbe4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q\" (UID: \"5822e703-13ed-4d49-8a12-2f51750ccbe4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"
Oct 30 04:44:59.271233 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.271106    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwgbh\" (UniqueName: \"kubernetes.io/projected/514edbce-f4ec-41b9-9b7a-e6879381720d-kube-api-access-hwgbh\") pod \"payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh\" (UID: \"514edbce-f4ec-41b9-9b7a-e6879381720d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"
Oct 30 04:44:59.271780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.271756    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514edbce-f4ec-41b9-9b7a-e6879381720d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh\" (UID: \"514edbce-f4ec-41b9-9b7a-e6879381720d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"
Oct 30 04:44:59.295734 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.295704    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwgbh\" (UniqueName: \"kubernetes.io/projected/514edbce-f4ec-41b9-9b7a-e6879381720d-kube-api-access-hwgbh\") pod \"payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh\" (UID: \"514edbce-f4ec-41b9-9b7a-e6879381720d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"
Oct 30 04:44:59.297778 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.297733    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr" podStartSLOduration=2.297719679 podStartE2EDuration="2.297719679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:59.296262369 +0000 UTC m=+2879.037685466" watchObservedRunningTime="2026-10-30 04:44:59.297719679 +0000 UTC m=+2879.039142770"
Oct 30 04:44:59.340511 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.340458    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h" podStartSLOduration=2.34043425 podStartE2EDuration="2.34043425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:59.337576019 +0000 UTC m=+2879.078999117" watchObservedRunningTime="2026-10-30 04:44:59.34043425 +0000 UTC m=+2879.081857346"
Oct 30 04:44:59.371943 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.371835    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5822e703-13ed-4d49-8a12-2f51750ccbe4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q\" (UID: \"5822e703-13ed-4d49-8a12-2f51750ccbe4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"
Oct 30 04:44:59.372161 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.372050    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lnwc\" (UniqueName: \"kubernetes.io/projected/5822e703-13ed-4d49-8a12-2f51750ccbe4-kube-api-access-6lnwc\") pod \"payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q\" (UID: \"5822e703-13ed-4d49-8a12-2f51750ccbe4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"
Oct 30 04:44:59.372715 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.372693    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5822e703-13ed-4d49-8a12-2f51750ccbe4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q\" (UID: \"5822e703-13ed-4d49-8a12-2f51750ccbe4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"
Oct 30 04:44:59.396635 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.396600    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lnwc\" (UniqueName: \"kubernetes.io/projected/5822e703-13ed-4d49-8a12-2f51750ccbe4-kube-api-access-6lnwc\") pod \"payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q\" (UID: \"5822e703-13ed-4d49-8a12-2f51750ccbe4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"
Oct 30 04:44:59.429597 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.429560    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"
Oct 30 04:44:59.536625 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.536586    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"
Oct 30 04:44:59.579289 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.572878    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"]
Oct 30 04:44:59.789198 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.789159    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"]
Oct 30 04:44:59.923003 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:44:59.922971    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"]
Oct 30 04:44:59.923471 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:44:59.923408    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5822e703_13ed_4d49_8a12_2f51750ccbe4.slice/crio-862639bc0729e0b1b7a3a3fd8a532b4f7bd8b6f040acb9d6d3f1989a35342180 WatchSource:0}: Error finding container 862639bc0729e0b1b7a3a3fd8a532b4f7bd8b6f040acb9d6d3f1989a35342180: Status 404 returned error can't find the container with id 862639bc0729e0b1b7a3a3fd8a532b4f7bd8b6f040acb9d6d3f1989a35342180
Oct 30 04:45:00.221015 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.220970    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"]
Oct 30 04:45:00.225627 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.225602    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"]
Oct 30 04:45:00.225757 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.225720    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"
Oct 30 04:45:00.229590 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.229569    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"]
Oct 30 04:45:00.229720 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.229702    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"
Oct 30 04:45:00.233478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.233455    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"]
Oct 30 04:45:00.233573 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.233564    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"
Oct 30 04:45:00.238070 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.238048    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"
Oct 30 04:45:00.246515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.246494    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:45:00.246808 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.246785    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh" event={"ID":"514edbce-f4ec-41b9-9b7a-e6879381720d","Type":"ContainerStarted","Data":"4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b"}
Oct 30 04:45:00.246894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.246831    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh" event={"ID":"514edbce-f4ec-41b9-9b7a-e6879381720d","Type":"ContainerStarted","Data":"a6dbacabbfc445b8625ea9a5609b3f38472e6b374a5b841acfc870d065a4c1c2"}
Oct 30 04:45:00.246894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.246873    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"
Oct 30 04:45:00.248818 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.248794    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q" event={"ID":"5822e703-13ed-4d49-8a12-2f51750ccbe4","Type":"ContainerStarted","Data":"13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5"}
Oct 30 04:45:00.248926 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.248840    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q" event={"ID":"5822e703-13ed-4d49-8a12-2f51750ccbe4","Type":"ContainerStarted","Data":"862639bc0729e0b1b7a3a3fd8a532b4f7bd8b6f040acb9d6d3f1989a35342180"}
Oct 30 04:45:00.248988 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.248950    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"
Oct 30 04:45:00.282469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.282188    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"]
Oct 30 04:45:00.286464 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.286437    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"]
Oct 30 04:45:00.294351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.294315    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x24z\" (UniqueName: \"kubernetes.io/projected/a7aa09dd-6974-44f5-9cc3-cbaba2223946-kube-api-access-4x24z\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824\" (UID: \"a7aa09dd-6974-44f5-9cc3-cbaba2223946\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"
Oct 30 04:45:00.294763 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.294741    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52403e2b-a240-452e-98f1-c577b1f770db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg\" (UID: \"52403e2b-a240-452e-98f1-c577b1f770db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"
Oct 30 04:45:00.295041 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.295008    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888925-m7sg5\" (UID: \"5a984c77-fa0d-4970-a1a3-55351ab6e7ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"
Oct 30 04:45:00.295598 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.295573    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8b785-0448-4ed4-b17a-38d43794b893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2\" (UID: \"a7d8b785-0448-4ed4-b17a-38d43794b893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"
Oct 30 04:45:00.295702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.295637    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsgrf\" (UniqueName: \"kubernetes.io/projected/a7d8b785-0448-4ed4-b17a-38d43794b893-kube-api-access-jsgrf\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2\" (UID: \"a7d8b785-0448-4ed4-b17a-38d43794b893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"
Oct 30 04:45:00.295702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.295666    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7aa09dd-6974-44f5-9cc3-cbaba2223946-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824\" (UID: \"a7aa09dd-6974-44f5-9cc3-cbaba2223946\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"
Oct 30 04:45:00.295810 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.295738    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9cvk\" (UniqueName: \"kubernetes.io/projected/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-kube-api-access-c9cvk\") pod \"maas-api-key-cleanup-29888925-m7sg5\" (UID: \"5a984c77-fa0d-4970-a1a3-55351ab6e7ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"
Oct 30 04:45:00.295810 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.295763    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgscn\" (UniqueName: \"kubernetes.io/projected/52403e2b-a240-452e-98f1-c577b1f770db-kube-api-access-rgscn\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg\" (UID: \"52403e2b-a240-452e-98f1-c577b1f770db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"
Oct 30 04:45:00.299053 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.299028    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"]
Oct 30 04:45:00.302071 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.302049    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"]
Oct 30 04:45:00.306971 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.306950    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"
Oct 30 04:45:00.312686 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.312659    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"]
Oct 30 04:45:00.321265 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.321241    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"]
Oct 30 04:45:00.321386 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.321336    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"
Oct 30 04:45:00.348091 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.345179    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"]
Oct 30 04:45:00.352044 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.352015    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"
Oct 30 04:45:00.358696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.358666    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"]
Oct 30 04:45:00.364873 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.364787    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"]
Oct 30 04:45:00.369270 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.369229    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"]
Oct 30 04:45:00.380697 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.380663    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"]
Oct 30 04:45:00.381424 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.381399    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"
Oct 30 04:45:00.396554 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.393894    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"
Oct 30 04:45:00.396554 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.395915    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"]
Oct 30 04:45:00.397703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.397117    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsgrf\" (UniqueName: \"kubernetes.io/projected/a7d8b785-0448-4ed4-b17a-38d43794b893-kube-api-access-jsgrf\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2\" (UID: \"a7d8b785-0448-4ed4-b17a-38d43794b893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"
Oct 30 04:45:00.399204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.399176    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7aa09dd-6974-44f5-9cc3-cbaba2223946-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824\" (UID: \"a7aa09dd-6974-44f5-9cc3-cbaba2223946\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"
Oct 30 04:45:00.399325 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.399228    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt\" (UID: \"2da7ba2d-6b50-4f01-8dc2-38552e35fd80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"
Oct 30 04:45:00.399325 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.399265    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac636d9-a1c1-4220-974f-46a2666f0005-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv\" (UID: \"eac636d9-a1c1-4220-974f-46a2666f0005\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"
Oct 30 04:45:00.399325 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.399309    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cvk\" (UniqueName: \"kubernetes.io/projected/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-kube-api-access-c9cvk\") pod \"maas-api-key-cleanup-29888925-m7sg5\" (UID: \"5a984c77-fa0d-4970-a1a3-55351ab6e7ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"
Oct 30 04:45:00.399524 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.399335    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgscn\" (UniqueName: \"kubernetes.io/projected/52403e2b-a240-452e-98f1-c577b1f770db-kube-api-access-rgscn\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg\" (UID: \"52403e2b-a240-452e-98f1-c577b1f770db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"
Oct 30 04:45:00.399524 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.399386    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x24z\" (UniqueName: \"kubernetes.io/projected/a7aa09dd-6974-44f5-9cc3-cbaba2223946-kube-api-access-4x24z\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824\" (UID: \"a7aa09dd-6974-44f5-9cc3-cbaba2223946\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"
Oct 30 04:45:00.400249 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.400032    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb2xh\" (UniqueName: \"kubernetes.io/projected/eac636d9-a1c1-4220-974f-46a2666f0005-kube-api-access-hb2xh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv\" (UID: \"eac636d9-a1c1-4220-974f-46a2666f0005\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"
Oct 30 04:45:00.400249 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.400100    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52403e2b-a240-452e-98f1-c577b1f770db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg\" (UID: \"52403e2b-a240-452e-98f1-c577b1f770db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"
Oct 30 04:45:00.400249 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.400145    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxkx\" (UniqueName: \"kubernetes.io/projected/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-kube-api-access-6cxkx\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt\" (UID: \"2da7ba2d-6b50-4f01-8dc2-38552e35fd80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"
Oct 30 04:45:00.400249 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.400170    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7aa09dd-6974-44f5-9cc3-cbaba2223946-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824\" (UID: \"a7aa09dd-6974-44f5-9cc3-cbaba2223946\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"
Oct 30 04:45:00.400249 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.400187    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888925-m7sg5\" (UID: \"5a984c77-fa0d-4970-a1a3-55351ab6e7ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"
Oct 30 04:45:00.400249 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.400237    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcf2c\" (UniqueName: \"kubernetes.io/projected/3abc0c26-3d30-4303-826f-7af0b072fc56-kube-api-access-jcf2c\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx\" (UID: \"3abc0c26-3d30-4303-826f-7af0b072fc56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"
Oct 30 04:45:00.400593 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.400320    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3abc0c26-3d30-4303-826f-7af0b072fc56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx\" (UID: \"3abc0c26-3d30-4303-826f-7af0b072fc56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"
Oct 30 04:45:00.400593 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.400370    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8b785-0448-4ed4-b17a-38d43794b893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2\" (UID: \"a7d8b785-0448-4ed4-b17a-38d43794b893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"
Oct 30 04:45:00.400937 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.400911    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888925-m7sg5\" (UID: \"5a984c77-fa0d-4970-a1a3-55351ab6e7ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"
Oct 30 04:45:00.401052 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.401032    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8b785-0448-4ed4-b17a-38d43794b893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2\" (UID: \"a7d8b785-0448-4ed4-b17a-38d43794b893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"
Oct 30 04:45:00.401596 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.401571    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52403e2b-a240-452e-98f1-c577b1f770db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg\" (UID: \"52403e2b-a240-452e-98f1-c577b1f770db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"
Oct 30 04:45:00.405963 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.405886    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh" podStartSLOduration=1.405869031 podStartE2EDuration="1.405869031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:00.390299301 +0000 UTC m=+2880.131722397" watchObservedRunningTime="2026-10-30 04:45:00.405869031 +0000 UTC m=+2880.147292130"
Oct 30 04:45:00.422392 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.416869    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"]
Oct 30 04:45:00.422392 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.416987    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"]
Oct 30 04:45:00.423917 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.423889    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsgrf\" (UniqueName: \"kubernetes.io/projected/a7d8b785-0448-4ed4-b17a-38d43794b893-kube-api-access-jsgrf\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2\" (UID: \"a7d8b785-0448-4ed4-b17a-38d43794b893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"
Oct 30 04:45:00.425604 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.425550    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9cvk\" (UniqueName: \"kubernetes.io/projected/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-kube-api-access-c9cvk\") pod \"maas-api-key-cleanup-29888925-m7sg5\" (UID: \"5a984c77-fa0d-4970-a1a3-55351ab6e7ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"
Oct 30 04:45:00.426045 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.426028    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgscn\" (UniqueName: \"kubernetes.io/projected/52403e2b-a240-452e-98f1-c577b1f770db-kube-api-access-rgscn\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg\" (UID: \"52403e2b-a240-452e-98f1-c577b1f770db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"
Oct 30 04:45:00.426453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.426438    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x24z\" (UniqueName: \"kubernetes.io/projected/a7aa09dd-6974-44f5-9cc3-cbaba2223946-kube-api-access-4x24z\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824\" (UID: \"a7aa09dd-6974-44f5-9cc3-cbaba2223946\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"
Oct 30 04:45:00.501434 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.501343    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxkx\" (UniqueName: \"kubernetes.io/projected/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-kube-api-access-6cxkx\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt\" (UID: \"2da7ba2d-6b50-4f01-8dc2-38552e35fd80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"
Oct 30 04:45:00.501434 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.501394    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcf2c\" (UniqueName: \"kubernetes.io/projected/3abc0c26-3d30-4303-826f-7af0b072fc56-kube-api-access-jcf2c\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx\" (UID: \"3abc0c26-3d30-4303-826f-7af0b072fc56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"
Oct 30 04:45:00.501434 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.501431    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldjg6\" (UniqueName: \"kubernetes.io/projected/25763423-975c-49f4-88a9-10b9e38015ea-kube-api-access-ldjg6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86\" (UID: \"25763423-975c-49f4-88a9-10b9e38015ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"
Oct 30 04:45:00.502028 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.501462    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbfr2\" (UniqueName: \"kubernetes.io/projected/900d8674-41c1-434e-86cf-562bf6ef0822-kube-api-access-mbfr2\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx\" (UID: \"900d8674-41c1-434e-86cf-562bf6ef0822\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"
Oct 30 04:45:00.502028 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.501521    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3abc0c26-3d30-4303-826f-7af0b072fc56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx\" (UID: \"3abc0c26-3d30-4303-826f-7af0b072fc56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"
Oct 30 04:45:00.502028 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.501566    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25763423-975c-49f4-88a9-10b9e38015ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86\" (UID: \"25763423-975c-49f4-88a9-10b9e38015ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"
Oct 30 04:45:00.502028 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.501628    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt\" (UID: \"2da7ba2d-6b50-4f01-8dc2-38552e35fd80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"
Oct 30 04:45:00.502028 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.501748    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac636d9-a1c1-4220-974f-46a2666f0005-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv\" (UID: \"eac636d9-a1c1-4220-974f-46a2666f0005\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"
Oct 30 04:45:00.502028 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.501822    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/900d8674-41c1-434e-86cf-562bf6ef0822-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx\" (UID: \"900d8674-41c1-434e-86cf-562bf6ef0822\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"
Oct 30 04:45:00.502028 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.501931    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb2xh\" (UniqueName: \"kubernetes.io/projected/eac636d9-a1c1-4220-974f-46a2666f0005-kube-api-access-hb2xh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv\" (UID: \"eac636d9-a1c1-4220-974f-46a2666f0005\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"
Oct 30 04:45:00.502462 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.502437    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt\" (UID: \"2da7ba2d-6b50-4f01-8dc2-38552e35fd80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"
Oct 30 04:45:00.502547 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.502526    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3abc0c26-3d30-4303-826f-7af0b072fc56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx\" (UID: \"3abc0c26-3d30-4303-826f-7af0b072fc56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"
Oct 30 04:45:00.502617 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.502595    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac636d9-a1c1-4220-974f-46a2666f0005-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv\" (UID: \"eac636d9-a1c1-4220-974f-46a2666f0005\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"
Oct 30 04:45:00.529827 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.529789    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxkx\" (UniqueName: \"kubernetes.io/projected/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-kube-api-access-6cxkx\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt\" (UID: \"2da7ba2d-6b50-4f01-8dc2-38552e35fd80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"
Oct 30 04:45:00.540221 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.534670    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb2xh\" (UniqueName: \"kubernetes.io/projected/eac636d9-a1c1-4220-974f-46a2666f0005-kube-api-access-hb2xh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv\" (UID: \"eac636d9-a1c1-4220-974f-46a2666f0005\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"
Oct 30 04:45:00.540221 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.535020    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcf2c\" (UniqueName: \"kubernetes.io/projected/3abc0c26-3d30-4303-826f-7af0b072fc56-kube-api-access-jcf2c\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx\" (UID: \"3abc0c26-3d30-4303-826f-7af0b072fc56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"
Oct 30 04:45:00.557344 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.557314    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:45:00.560495 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.560467    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"
Oct 30 04:45:00.562705 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.562682    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"
Oct 30 04:45:00.564158 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.564125    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"
Oct 30 04:45:00.569185 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.569158    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"
Oct 30 04:45:00.603869 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.603817    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/900d8674-41c1-434e-86cf-562bf6ef0822-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx\" (UID: \"900d8674-41c1-434e-86cf-562bf6ef0822\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"
Oct 30 04:45:00.604033 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.603980    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjg6\" (UniqueName: \"kubernetes.io/projected/25763423-975c-49f4-88a9-10b9e38015ea-kube-api-access-ldjg6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86\" (UID: \"25763423-975c-49f4-88a9-10b9e38015ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"
Oct 30 04:45:00.604033 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.604015    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbfr2\" (UniqueName: \"kubernetes.io/projected/900d8674-41c1-434e-86cf-562bf6ef0822-kube-api-access-mbfr2\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx\" (UID: \"900d8674-41c1-434e-86cf-562bf6ef0822\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"
Oct 30 04:45:00.604149 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.604130    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25763423-975c-49f4-88a9-10b9e38015ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86\" (UID: \"25763423-975c-49f4-88a9-10b9e38015ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"
Oct 30 04:45:00.606222 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.605700    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/900d8674-41c1-434e-86cf-562bf6ef0822-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx\" (UID: \"900d8674-41c1-434e-86cf-562bf6ef0822\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"
Oct 30 04:45:00.607673 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.607278    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q" podStartSLOduration=1.6072636679999999 podStartE2EDuration="1.607263668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:00.606470393 +0000 UTC m=+2880.347893493" watchObservedRunningTime="2026-10-30 04:45:00.607263668 +0000 UTC m=+2880.348686766"
Oct 30 04:45:00.607923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.607711    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25763423-975c-49f4-88a9-10b9e38015ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86\" (UID: \"25763423-975c-49f4-88a9-10b9e38015ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"
Oct 30 04:45:00.631005 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.630905    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"
Oct 30 04:45:00.634510 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.633923    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldjg6\" (UniqueName: \"kubernetes.io/projected/25763423-975c-49f4-88a9-10b9e38015ea-kube-api-access-ldjg6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86\" (UID: \"25763423-975c-49f4-88a9-10b9e38015ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"
Oct 30 04:45:00.634510 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.634452    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbfr2\" (UniqueName: \"kubernetes.io/projected/900d8674-41c1-434e-86cf-562bf6ef0822-kube-api-access-mbfr2\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx\" (UID: \"900d8674-41c1-434e-86cf-562bf6ef0822\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"
Oct 30 04:45:00.685055 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.668039    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"
Oct 30 04:45:00.689420 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.685875    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"
Oct 30 04:45:00.714946 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.714720    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"
Oct 30 04:45:00.726952 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:00.726786    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"
Oct 30 04:45:01.097003 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:45:01.096954    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52403e2b_a240_452e_98f1_c577b1f770db.slice/crio-5428020108eed706e392d9aba5e2f6ab313ef78f79f12338486a6d8102ae3132 WatchSource:0}: Error finding container 5428020108eed706e392d9aba5e2f6ab313ef78f79f12338486a6d8102ae3132: Status 404 returned error can't find the container with id 5428020108eed706e392d9aba5e2f6ab313ef78f79f12338486a6d8102ae3132
Oct 30 04:45:01.098676 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:01.098640    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"]
Oct 30 04:45:01.132886 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:01.132861    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"]
Oct 30 04:45:01.134692 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:45:01.134661    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a984c77_fa0d_4970_a1a3_55351ab6e7ee.slice/crio-238663de760cc44383be1e2a41b182115c9b0547aafba384ea4e0320e5bba8f1 WatchSource:0}: Error finding container 238663de760cc44383be1e2a41b182115c9b0547aafba384ea4e0320e5bba8f1: Status 404 returned error can't find the container with id 238663de760cc44383be1e2a41b182115c9b0547aafba384ea4e0320e5bba8f1
Oct 30 04:45:01.139423 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:45:01.139397    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d8b785_0448_4ed4_b17a_38d43794b893.slice/crio-d49bc9abccf557957bc1a39ca491c9b1d499b3f6a8419ca0f5af4d2df948a4cf WatchSource:0}: Error finding container d49bc9abccf557957bc1a39ca491c9b1d499b3f6a8419ca0f5af4d2df948a4cf: Status 404 returned error can't find the container with id d49bc9abccf557957bc1a39ca491c9b1d499b3f6a8419ca0f5af4d2df948a4cf
Oct 30 04:45:01.141050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:01.141022    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"]
Oct 30 04:45:01.177679 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:01.177649    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"]
Oct 30 04:45:01.236016 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:01.235987    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"]
Oct 30 04:45:01.271474 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:01.271440    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5" event={"ID":"5a984c77-fa0d-4970-a1a3-55351ab6e7ee","Type":"ContainerStarted","Data":"238663de760cc44383be1e2a41b182115c9b0547aafba384ea4e0320e5bba8f1"}
Oct 30 04:45:01.272970 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:01.272941    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2" event={"ID":"a7d8b785-0448-4ed4-b17a-38d43794b893","Type":"ContainerStarted","Data":"d49bc9abccf557957bc1a39ca491c9b1d499b3f6a8419ca0f5af4d2df948a4cf"}
Oct 30 04:45:01.275128 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:45:01.275061    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7aa09dd_6974_44f5_9cc3_cbaba2223946.slice/crio-8c8d0baefd7c6c0da788d1288986fe2d746c2ae2f0116f1d61a4ff3beed370fe WatchSource:0}: Error finding container 8c8d0baefd7c6c0da788d1288986fe2d746c2ae2f0116f1d61a4ff3beed370fe: Status 404 returned error can't find the container with id 8c8d0baefd7c6c0da788d1288986fe2d746c2ae2f0116f1d61a4ff3beed370fe
Oct 30 04:45:01.275227 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:01.275154    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg" event={"ID":"52403e2b-a240-452e-98f1-c577b1f770db","Type":"ContainerStarted","Data":"5428020108eed706e392d9aba5e2f6ab313ef78f79f12338486a6d8102ae3132"}
Oct 30 04:45:01.287088 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:01.287045    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"]
Oct 30 04:45:01.310064 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:01.310036    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"]
Oct 30 04:45:01.326331 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:01.326306    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"]
Oct 30 04:45:01.332386 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:01.332361    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"]
Oct 30 04:45:01.371030 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:45:01.370983    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abc0c26_3d30_4303_826f_7af0b072fc56.slice/crio-cddc501b06e06353b5eb307de328e049528d909f8b954cab51dcfd503a0b4c89 WatchSource:0}: Error finding container cddc501b06e06353b5eb307de328e049528d909f8b954cab51dcfd503a0b4c89: Status 404 returned error can't find the container with id cddc501b06e06353b5eb307de328e049528d909f8b954cab51dcfd503a0b4c89
Oct 30 04:45:01.371722 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:45:01.371698    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac636d9_a1c1_4220_974f_46a2666f0005.slice/crio-3531c6d767667636b3007ca262d6fdb56b543ec8b539f5dbe80cbc9ed6eee015 WatchSource:0}: Error finding container 3531c6d767667636b3007ca262d6fdb56b543ec8b539f5dbe80cbc9ed6eee015: Status 404 returned error can't find the container with id 3531c6d767667636b3007ca262d6fdb56b543ec8b539f5dbe80cbc9ed6eee015
Oct 30 04:45:01.372631 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:45:01.372550    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900d8674_41c1_434e_86cf_562bf6ef0822.slice/crio-b71593fae05cadc9d3d128434387a0ca84bb1edd4416d57e7065406aea9fac36 WatchSource:0}: Error finding container b71593fae05cadc9d3d128434387a0ca84bb1edd4416d57e7065406aea9fac36: Status 404 returned error can't find the container with id b71593fae05cadc9d3d128434387a0ca84bb1edd4416d57e7065406aea9fac36
Oct 30 04:45:01.374114 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:45:01.374060    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25763423_975c_49f4_88a9_10b9e38015ea.slice/crio-6eccbd6a618b063f02eb80c151687ee1cf866297d0bd64251110b8ffdda0792d WatchSource:0}: Error finding container 6eccbd6a618b063f02eb80c151687ee1cf866297d0bd64251110b8ffdda0792d: Status 404 returned error can't find the container with id 6eccbd6a618b063f02eb80c151687ee1cf866297d0bd64251110b8ffdda0792d
Oct 30 04:45:02.285169 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.285051    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5" event={"ID":"5a984c77-fa0d-4970-a1a3-55351ab6e7ee","Type":"ContainerStarted","Data":"61b0bda0ce70d1beaa136ae728595a4d0a51942ea171b0ebe9bbf6998ed6e7f1"}
Oct 30 04:45:02.287307 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.287274    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2" event={"ID":"a7d8b785-0448-4ed4-b17a-38d43794b893","Type":"ContainerStarted","Data":"dc73ea2158dc4a3bc99d38899cca578a0f3d8349ea5ec1fba8230e7898151743"}
Oct 30 04:45:02.289130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.289105    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx" event={"ID":"3abc0c26-3d30-4303-826f-7af0b072fc56","Type":"ContainerStarted","Data":"b0c7ee16b4b94e43832b587fed65ec211c28a72421e5051996c4c793f3900a1e"}
Oct 30 04:45:02.289244 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.289150    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx" event={"ID":"3abc0c26-3d30-4303-826f-7af0b072fc56","Type":"ContainerStarted","Data":"cddc501b06e06353b5eb307de328e049528d909f8b954cab51dcfd503a0b4c89"}
Oct 30 04:45:02.291276 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.291251    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt" event={"ID":"2da7ba2d-6b50-4f01-8dc2-38552e35fd80","Type":"ContainerStarted","Data":"95e3133ecf0e8c1d3bcba6605c0e88639436e72df6e4e55906a139923f4355a5"}
Oct 30 04:45:02.291783 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.291760    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt" event={"ID":"2da7ba2d-6b50-4f01-8dc2-38552e35fd80","Type":"ContainerStarted","Data":"aa0c014402907c49805f0e03003499b5597732b0fb2ded007f3806d857c873a2"}
Oct 30 04:45:02.293298 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.293272    5072 generic.go:356] "Generic (PLEG): container finished" podID="52403e2b-a240-452e-98f1-c577b1f770db" containerID="e7d1eaae9d642e421a629885cb5bee68e0820bbc5b782b7174f2883fb508f90d" exitCode=60
Oct 30 04:45:02.293413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.293349    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg" event={"ID":"52403e2b-a240-452e-98f1-c577b1f770db","Type":"ContainerDied","Data":"e7d1eaae9d642e421a629885cb5bee68e0820bbc5b782b7174f2883fb508f90d"}
Oct 30 04:45:02.293791 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.293544    5072 scope.go:122] "RemoveContainer" containerID="e7d1eaae9d642e421a629885cb5bee68e0820bbc5b782b7174f2883fb508f90d"
Oct 30 04:45:02.295915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.295890    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824" event={"ID":"a7aa09dd-6974-44f5-9cc3-cbaba2223946","Type":"ContainerStarted","Data":"0bb885803bbaedc8077886d5bc68dd693c7eaf40e65f9e425d950297fcb02c98"}
Oct 30 04:45:02.296019 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.295929    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824" event={"ID":"a7aa09dd-6974-44f5-9cc3-cbaba2223946","Type":"ContainerStarted","Data":"8c8d0baefd7c6c0da788d1288986fe2d746c2ae2f0116f1d61a4ff3beed370fe"}
Oct 30 04:45:02.304936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.304903    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv" event={"ID":"eac636d9-a1c1-4220-974f-46a2666f0005","Type":"ContainerStarted","Data":"fd79d9fc2b1009feaa3592b1f6b6275a748d0b8dbe6d1297709094b87f210ac7"}
Oct 30 04:45:02.305098 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.304965    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv" event={"ID":"eac636d9-a1c1-4220-974f-46a2666f0005","Type":"ContainerStarted","Data":"3531c6d767667636b3007ca262d6fdb56b543ec8b539f5dbe80cbc9ed6eee015"}
Oct 30 04:45:02.306642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.306618    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx" event={"ID":"900d8674-41c1-434e-86cf-562bf6ef0822","Type":"ContainerStarted","Data":"61e4202e68d02402d22f301513a0c9c0bed4a2a66c4e2de8ef1193ed9b0fed73"}
Oct 30 04:45:02.306749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.306666    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx" event={"ID":"900d8674-41c1-434e-86cf-562bf6ef0822","Type":"ContainerStarted","Data":"b71593fae05cadc9d3d128434387a0ca84bb1edd4416d57e7065406aea9fac36"}
Oct 30 04:45:02.308399 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.308377    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86" event={"ID":"25763423-975c-49f4-88a9-10b9e38015ea","Type":"ContainerStarted","Data":"d67951118cd63b2aa33fabbbe4a7a421c21c8c058f2cef65168664024989eca0"}
Oct 30 04:45:02.308499 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.308420    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86" event={"ID":"25763423-975c-49f4-88a9-10b9e38015ea","Type":"ContainerStarted","Data":"6eccbd6a618b063f02eb80c151687ee1cf866297d0bd64251110b8ffdda0792d"}
Oct 30 04:45:02.332292 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.329568    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5" podStartSLOduration=2.329547757 podStartE2EDuration="2.329547757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:02.329295849 +0000 UTC m=+2882.070718948" watchObservedRunningTime="2026-10-30 04:45:02.329547757 +0000 UTC m=+2882.070970854"
Oct 30 04:45:02.415543 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.415478    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt" podStartSLOduration=2.415458765 podStartE2EDuration="2.415458765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:02.372779917 +0000 UTC m=+2882.114203015" watchObservedRunningTime="2026-10-30 04:45:02.415458765 +0000 UTC m=+2882.156881862"
Oct 30 04:45:02.418464 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.418416    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx" podStartSLOduration=2.418402842 podStartE2EDuration="2.418402842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:02.415000719 +0000 UTC m=+2882.156423817" watchObservedRunningTime="2026-10-30 04:45:02.418402842 +0000 UTC m=+2882.159825937"
Oct 30 04:45:02.464357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.464272    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824" podStartSLOduration=2.46425348 podStartE2EDuration="2.46425348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:02.461302506 +0000 UTC m=+2882.202725603" watchObservedRunningTime="2026-10-30 04:45:02.46425348 +0000 UTC m=+2882.205676581"
Oct 30 04:45:02.504060 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.503998    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx" podStartSLOduration=2.503979124 podStartE2EDuration="2.503979124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:02.502132619 +0000 UTC m=+2882.243555717" watchObservedRunningTime="2026-10-30 04:45:02.503979124 +0000 UTC m=+2882.245402223"
Oct 30 04:45:02.545852 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.545735    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86" podStartSLOduration=2.5457171929999998 podStartE2EDuration="2.545717193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:02.542690989 +0000 UTC m=+2882.284114088" watchObservedRunningTime="2026-10-30 04:45:02.545717193 +0000 UTC m=+2882.287140290"
Oct 30 04:45:02.637975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.637908    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2" podStartSLOduration=2.63789132 podStartE2EDuration="2.63789132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:02.63459309 +0000 UTC m=+2882.376016186" watchObservedRunningTime="2026-10-30 04:45:02.63789132 +0000 UTC m=+2882.379314417"
Oct 30 04:45:02.680994 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.680928    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv" podStartSLOduration=2.680906276 podStartE2EDuration="2.680906276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:02.677394999 +0000 UTC m=+2882.418818098" watchObservedRunningTime="2026-10-30 04:45:02.680906276 +0000 UTC m=+2882.422329374"
Oct 30 04:45:02.894255 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.894219    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"]
Oct 30 04:45:02.898779 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.898752    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"
Oct 30 04:45:02.915502 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.915425    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-05bee8\""
Oct 30 04:45:02.915502 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.915472    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-05bee8-dockercfg-57b4r\""
Oct 30 04:45:02.916786 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.916755    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"]
Oct 30 04:45:02.940904 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.940869    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23dd655b-f213-4e21-8586-7413a7bea249-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6\" (UID: \"23dd655b-f213-4e21-8586-7413a7bea249\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"
Oct 30 04:45:02.941239 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:02.941212    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvsp\" (UniqueName: \"kubernetes.io/projected/23dd655b-f213-4e21-8586-7413a7bea249-kube-api-access-prvsp\") pod \"payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6\" (UID: \"23dd655b-f213-4e21-8586-7413a7bea249\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"
Oct 30 04:45:03.019951 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.019911    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"]
Oct 30 04:45:03.035514 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.035467    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"]
Oct 30 04:45:03.035672 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.035598    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"
Oct 30 04:45:03.042065 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.042026    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvsp\" (UniqueName: \"kubernetes.io/projected/23dd655b-f213-4e21-8586-7413a7bea249-kube-api-access-prvsp\") pod \"payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6\" (UID: \"23dd655b-f213-4e21-8586-7413a7bea249\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"
Oct 30 04:45:03.042065 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.042107    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75405bbd-d694-4d30-9c5b-122843efcd55-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb\" (UID: \"75405bbd-d694-4d30-9c5b-122843efcd55\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"
Oct 30 04:45:03.042347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.042136    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23dd655b-f213-4e21-8586-7413a7bea249-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6\" (UID: \"23dd655b-f213-4e21-8586-7413a7bea249\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"
Oct 30 04:45:03.042347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.042162    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7hfz\" (UniqueName: \"kubernetes.io/projected/75405bbd-d694-4d30-9c5b-122843efcd55-kube-api-access-h7hfz\") pod \"payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb\" (UID: \"75405bbd-d694-4d30-9c5b-122843efcd55\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"
Oct 30 04:45:03.043388 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.043361    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23dd655b-f213-4e21-8586-7413a7bea249-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6\" (UID: \"23dd655b-f213-4e21-8586-7413a7bea249\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"
Oct 30 04:45:03.069197 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.069166    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvsp\" (UniqueName: \"kubernetes.io/projected/23dd655b-f213-4e21-8586-7413a7bea249-kube-api-access-prvsp\") pod \"payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6\" (UID: \"23dd655b-f213-4e21-8586-7413a7bea249\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"
Oct 30 04:45:03.143475 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.143377    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75405bbd-d694-4d30-9c5b-122843efcd55-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb\" (UID: \"75405bbd-d694-4d30-9c5b-122843efcd55\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"
Oct 30 04:45:03.143475 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.143420    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7hfz\" (UniqueName: \"kubernetes.io/projected/75405bbd-d694-4d30-9c5b-122843efcd55-kube-api-access-h7hfz\") pod \"payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb\" (UID: \"75405bbd-d694-4d30-9c5b-122843efcd55\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"
Oct 30 04:45:03.144143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.144118    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75405bbd-d694-4d30-9c5b-122843efcd55-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb\" (UID: \"75405bbd-d694-4d30-9c5b-122843efcd55\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"
Oct 30 04:45:03.167942 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.167911    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7hfz\" (UniqueName: \"kubernetes.io/projected/75405bbd-d694-4d30-9c5b-122843efcd55-kube-api-access-h7hfz\") pod \"payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb\" (UID: \"75405bbd-d694-4d30-9c5b-122843efcd55\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"
Oct 30 04:45:03.245147 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.245110    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"
Oct 30 04:45:03.325036 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.325001    5072 generic.go:356] "Generic (PLEG): container finished" podID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" containerID="95e3133ecf0e8c1d3bcba6605c0e88639436e72df6e4e55906a139923f4355a5" exitCode=60
Oct 30 04:45:03.325508 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.325129    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt" event={"ID":"2da7ba2d-6b50-4f01-8dc2-38552e35fd80","Type":"ContainerDied","Data":"95e3133ecf0e8c1d3bcba6605c0e88639436e72df6e4e55906a139923f4355a5"}
Oct 30 04:45:03.325508 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.325397    5072 scope.go:122] "RemoveContainer" containerID="95e3133ecf0e8c1d3bcba6605c0e88639436e72df6e4e55906a139923f4355a5"
Oct 30 04:45:03.341445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.341376    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg" event={"ID":"52403e2b-a240-452e-98f1-c577b1f770db","Type":"ContainerStarted","Data":"4ab4a8708611fc9945cdae4b8e6f171123393f7333a98f143f049c5e55061ae3"}
Oct 30 04:45:03.343838 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.343804    5072 generic.go:356] "Generic (PLEG): container finished" podID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" containerID="0bb885803bbaedc8077886d5bc68dd693c7eaf40e65f9e425d950297fcb02c98" exitCode=60
Oct 30 04:45:03.344095 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.343994    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824" event={"ID":"a7aa09dd-6974-44f5-9cc3-cbaba2223946","Type":"ContainerDied","Data":"0bb885803bbaedc8077886d5bc68dd693c7eaf40e65f9e425d950297fcb02c98"}
Oct 30 04:45:03.344291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.344256    5072 scope.go:122] "RemoveContainer" containerID="0bb885803bbaedc8077886d5bc68dd693c7eaf40e65f9e425d950297fcb02c98"
Oct 30 04:45:03.346839 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.346815    5072 generic.go:356] "Generic (PLEG): container finished" podID="eac636d9-a1c1-4220-974f-46a2666f0005" containerID="fd79d9fc2b1009feaa3592b1f6b6275a748d0b8dbe6d1297709094b87f210ac7" exitCode=60
Oct 30 04:45:03.346958 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.346905    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv" event={"ID":"eac636d9-a1c1-4220-974f-46a2666f0005","Type":"ContainerDied","Data":"fd79d9fc2b1009feaa3592b1f6b6275a748d0b8dbe6d1297709094b87f210ac7"}
Oct 30 04:45:03.347147 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.347128    5072 scope.go:122] "RemoveContainer" containerID="fd79d9fc2b1009feaa3592b1f6b6275a748d0b8dbe6d1297709094b87f210ac7"
Oct 30 04:45:03.349916 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.349833    5072 generic.go:356] "Generic (PLEG): container finished" podID="900d8674-41c1-434e-86cf-562bf6ef0822" containerID="61e4202e68d02402d22f301513a0c9c0bed4a2a66c4e2de8ef1193ed9b0fed73" exitCode=60
Oct 30 04:45:03.349916 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.349904    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx" event={"ID":"900d8674-41c1-434e-86cf-562bf6ef0822","Type":"ContainerDied","Data":"61e4202e68d02402d22f301513a0c9c0bed4a2a66c4e2de8ef1193ed9b0fed73"}
Oct 30 04:45:03.350222 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.350205    5072 scope.go:122] "RemoveContainer" containerID="61e4202e68d02402d22f301513a0c9c0bed4a2a66c4e2de8ef1193ed9b0fed73"
Oct 30 04:45:03.356752 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.356729    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"
Oct 30 04:45:03.362734 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.359011    5072 generic.go:356] "Generic (PLEG): container finished" podID="25763423-975c-49f4-88a9-10b9e38015ea" containerID="d67951118cd63b2aa33fabbbe4a7a421c21c8c058f2cef65168664024989eca0" exitCode=60
Oct 30 04:45:03.362734 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.359139    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86" event={"ID":"25763423-975c-49f4-88a9-10b9e38015ea","Type":"ContainerDied","Data":"d67951118cd63b2aa33fabbbe4a7a421c21c8c058f2cef65168664024989eca0"}
Oct 30 04:45:03.362734 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.362339    5072 scope.go:122] "RemoveContainer" containerID="d67951118cd63b2aa33fabbbe4a7a421c21c8c058f2cef65168664024989eca0"
Oct 30 04:45:03.365821 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.365792    5072 generic.go:356] "Generic (PLEG): container finished" podID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" containerID="61b0bda0ce70d1beaa136ae728595a4d0a51942ea171b0ebe9bbf6998ed6e7f1" exitCode=60
Oct 30 04:45:03.365921 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.365885    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5" event={"ID":"5a984c77-fa0d-4970-a1a3-55351ab6e7ee","Type":"ContainerDied","Data":"61b0bda0ce70d1beaa136ae728595a4d0a51942ea171b0ebe9bbf6998ed6e7f1"}
Oct 30 04:45:03.366310 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.366289    5072 scope.go:122] "RemoveContainer" containerID="61b0bda0ce70d1beaa136ae728595a4d0a51942ea171b0ebe9bbf6998ed6e7f1"
Oct 30 04:45:03.369657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.369617    5072 generic.go:356] "Generic (PLEG): container finished" podID="a7d8b785-0448-4ed4-b17a-38d43794b893" containerID="dc73ea2158dc4a3bc99d38899cca578a0f3d8349ea5ec1fba8230e7898151743" exitCode=60
Oct 30 04:45:03.371360 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.369752    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2" event={"ID":"a7d8b785-0448-4ed4-b17a-38d43794b893","Type":"ContainerDied","Data":"dc73ea2158dc4a3bc99d38899cca578a0f3d8349ea5ec1fba8230e7898151743"}
Oct 30 04:45:03.371360 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.370351    5072 scope.go:122] "RemoveContainer" containerID="dc73ea2158dc4a3bc99d38899cca578a0f3d8349ea5ec1fba8230e7898151743"
Oct 30 04:45:03.376638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.376614    5072 generic.go:356] "Generic (PLEG): container finished" podID="3abc0c26-3d30-4303-826f-7af0b072fc56" containerID="b0c7ee16b4b94e43832b587fed65ec211c28a72421e5051996c4c793f3900a1e" exitCode=60
Oct 30 04:45:03.376742 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.376678    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx" event={"ID":"3abc0c26-3d30-4303-826f-7af0b072fc56","Type":"ContainerDied","Data":"b0c7ee16b4b94e43832b587fed65ec211c28a72421e5051996c4c793f3900a1e"}
Oct 30 04:45:03.376929 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.376913    5072 scope.go:122] "RemoveContainer" containerID="b0c7ee16b4b94e43832b587fed65ec211c28a72421e5051996c4c793f3900a1e"
Oct 30 04:45:03.463211 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.463129    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg" podStartSLOduration=3.463106397 podStartE2EDuration="3.463106397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:03.413267024 +0000 UTC m=+2883.154690120" watchObservedRunningTime="2026-10-30 04:45:03.463106397 +0000 UTC m=+2883.204529494"
Oct 30 04:45:03.559004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.558978    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"]
Oct 30 04:45:03.564300 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:45:03.564269    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23dd655b_f213_4e21_8586_7413a7bea249.slice/crio-e1f201cb4285f952ba10549aec7b4e4ca52555a428c3f2ed91d4a5c601034c21 WatchSource:0}: Error finding container e1f201cb4285f952ba10549aec7b4e4ca52555a428c3f2ed91d4a5c601034c21: Status 404 returned error can't find the container with id e1f201cb4285f952ba10549aec7b4e4ca52555a428c3f2ed91d4a5c601034c21
Oct 30 04:45:03.728253 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:03.728223    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"]
Oct 30 04:45:03.730683 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:45:03.730655    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75405bbd_d694_4d30_9c5b_122843efcd55.slice/crio-4d90d13e55daa7958cfc3ddd09055e62989beefe6cc00bdf834dfc7af00f50e0 WatchSource:0}: Error finding container 4d90d13e55daa7958cfc3ddd09055e62989beefe6cc00bdf834dfc7af00f50e0: Status 404 returned error can't find the container with id 4d90d13e55daa7958cfc3ddd09055e62989beefe6cc00bdf834dfc7af00f50e0
Oct 30 04:45:04.387547 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.387510    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824" event={"ID":"a7aa09dd-6974-44f5-9cc3-cbaba2223946","Type":"ContainerStarted","Data":"d9d726a04cdeff0524bc23af60351de8ef1b9deba9d71bddc3260abac9a2aacb"}
Oct 30 04:45:04.389786 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.389756    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv" event={"ID":"eac636d9-a1c1-4220-974f-46a2666f0005","Type":"ContainerStarted","Data":"438f102abbf5b96371977baa58368ca94d546b32689b2b0173f75bd7f364172c"}
Oct 30 04:45:04.391538 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.391514    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb" event={"ID":"75405bbd-d694-4d30-9c5b-122843efcd55","Type":"ContainerStarted","Data":"5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160"}
Oct 30 04:45:04.391665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.391559    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"
Oct 30 04:45:04.391665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.391579    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb" event={"ID":"75405bbd-d694-4d30-9c5b-122843efcd55","Type":"ContainerStarted","Data":"4d90d13e55daa7958cfc3ddd09055e62989beefe6cc00bdf834dfc7af00f50e0"}
Oct 30 04:45:04.393273 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.393239    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6" event={"ID":"23dd655b-f213-4e21-8586-7413a7bea249","Type":"ContainerStarted","Data":"388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44"}
Oct 30 04:45:04.393373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.393284    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6" event={"ID":"23dd655b-f213-4e21-8586-7413a7bea249","Type":"ContainerStarted","Data":"e1f201cb4285f952ba10549aec7b4e4ca52555a428c3f2ed91d4a5c601034c21"}
Oct 30 04:45:04.393373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.393313    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"
Oct 30 04:45:04.395209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.395188    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx" event={"ID":"900d8674-41c1-434e-86cf-562bf6ef0822","Type":"ContainerStarted","Data":"b3a512fa5607e8d6526aecf62dbcfe5481ed0c9d7e5c050b42e7d3c912a4f019"}
Oct 30 04:45:04.397132 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.397103    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86" event={"ID":"25763423-975c-49f4-88a9-10b9e38015ea","Type":"ContainerStarted","Data":"5174075df0ef240cddde365a9885a3889f4649bb1b1659de3b2395804a2a2e75"}
Oct 30 04:45:04.399124 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.399101    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5" event={"ID":"5a984c77-fa0d-4970-a1a3-55351ab6e7ee","Type":"ContainerStarted","Data":"1a98182e71eade579f14e19fbfc0cc134b767e2dc86d00f97e6d530e80f658cd"}
Oct 30 04:45:04.406966 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.406925    5072 generic.go:356] "Generic (PLEG): container finished" podID="52403e2b-a240-452e-98f1-c577b1f770db" containerID="4ab4a8708611fc9945cdae4b8e6f171123393f7333a98f143f049c5e55061ae3" exitCode=60
Oct 30 04:45:04.407254 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.407237    5072 scope.go:122] "RemoveContainer" containerID="4ab4a8708611fc9945cdae4b8e6f171123393f7333a98f143f049c5e55061ae3"
Oct 30 04:45:04.407461 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:04.407441    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg_odh-ai-gateway-infra(52403e2b-a240-452e-98f1-c577b1f770db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg" podUID="52403e2b-a240-452e-98f1-c577b1f770db"
Oct 30 04:45:04.409437 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.409413    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2" event={"ID":"a7d8b785-0448-4ed4-b17a-38d43794b893","Type":"ContainerStarted","Data":"bd889596a85c0e95d602798085895f92275c8d1801a88fe2d74924883bf6df81"}
Oct 30 04:45:04.409515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.409461    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx" event={"ID":"3abc0c26-3d30-4303-826f-7af0b072fc56","Type":"ContainerStarted","Data":"7b82bfddaca0b6d5aa2c6b6f03be9f990329c2578801b3b7c4203d2759711d9c"}
Oct 30 04:45:04.409515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.409485    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt" event={"ID":"2da7ba2d-6b50-4f01-8dc2-38552e35fd80","Type":"ContainerStarted","Data":"537be6e231df499178ed06407302a00f98910f7fc39a1b1ec1bc90fef4391850"}
Oct 30 04:45:04.409515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.409505    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg" event={"ID":"52403e2b-a240-452e-98f1-c577b1f770db","Type":"ContainerDied","Data":"4ab4a8708611fc9945cdae4b8e6f171123393f7333a98f143f049c5e55061ae3"}
Oct 30 04:45:04.409652 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.409528    5072 scope.go:122] "RemoveContainer" containerID="e7d1eaae9d642e421a629885cb5bee68e0820bbc5b782b7174f2883fb508f90d"
Oct 30 04:45:04.693568 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.693508    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb" podStartSLOduration=2.693492826 podStartE2EDuration="2.693492826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:04.690603157 +0000 UTC m=+2884.432026260" watchObservedRunningTime="2026-10-30 04:45:04.693492826 +0000 UTC m=+2884.434916147"
Oct 30 04:45:04.903985 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:04.903931    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6" podStartSLOduration=2.9039150080000002 podStartE2EDuration="2.903915008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:04.902770149 +0000 UTC m=+2884.644193246" watchObservedRunningTime="2026-10-30 04:45:04.903915008 +0000 UTC m=+2884.645338108"
Oct 30 04:45:05.337403 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.337352    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26" podUID="fe067687-49f8-4781-9c02-a2b47ea35293" containerName="main" containerID="cri-o://3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334" gracePeriod=45
Oct 30 04:45:05.417114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.417059    5072 generic.go:356] "Generic (PLEG): container finished" podID="eac636d9-a1c1-4220-974f-46a2666f0005" containerID="438f102abbf5b96371977baa58368ca94d546b32689b2b0173f75bd7f364172c" exitCode=60
Oct 30 04:45:05.417565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.417117    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv" event={"ID":"eac636d9-a1c1-4220-974f-46a2666f0005","Type":"ContainerDied","Data":"438f102abbf5b96371977baa58368ca94d546b32689b2b0173f75bd7f364172c"}
Oct 30 04:45:05.417565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.417204    5072 scope.go:122] "RemoveContainer" containerID="fd79d9fc2b1009feaa3592b1f6b6275a748d0b8dbe6d1297709094b87f210ac7"
Oct 30 04:45:05.417565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.417491    5072 scope.go:122] "RemoveContainer" containerID="438f102abbf5b96371977baa58368ca94d546b32689b2b0173f75bd7f364172c"
Oct 30 04:45:05.417805 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:05.417785    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv_odh-ai-gateway-infra(eac636d9-a1c1-4220-974f-46a2666f0005)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv" podUID="eac636d9-a1c1-4220-974f-46a2666f0005"
Oct 30 04:45:05.419499 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.419476    5072 generic.go:356] "Generic (PLEG): container finished" podID="900d8674-41c1-434e-86cf-562bf6ef0822" containerID="b3a512fa5607e8d6526aecf62dbcfe5481ed0c9d7e5c050b42e7d3c912a4f019" exitCode=60
Oct 30 04:45:05.419611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.419551    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx" event={"ID":"900d8674-41c1-434e-86cf-562bf6ef0822","Type":"ContainerDied","Data":"b3a512fa5607e8d6526aecf62dbcfe5481ed0c9d7e5c050b42e7d3c912a4f019"}
Oct 30 04:45:05.419892 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.419822    5072 scope.go:122] "RemoveContainer" containerID="b3a512fa5607e8d6526aecf62dbcfe5481ed0c9d7e5c050b42e7d3c912a4f019"
Oct 30 04:45:05.420142 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:05.420123    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx_odh-ai-gateway-infra(900d8674-41c1-434e-86cf-562bf6ef0822)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx" podUID="900d8674-41c1-434e-86cf-562bf6ef0822"
Oct 30 04:45:05.421392 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.421373    5072 generic.go:356] "Generic (PLEG): container finished" podID="25763423-975c-49f4-88a9-10b9e38015ea" containerID="5174075df0ef240cddde365a9885a3889f4649bb1b1659de3b2395804a2a2e75" exitCode=60
Oct 30 04:45:05.421465 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.421448    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86" event={"ID":"25763423-975c-49f4-88a9-10b9e38015ea","Type":"ContainerDied","Data":"5174075df0ef240cddde365a9885a3889f4649bb1b1659de3b2395804a2a2e75"}
Oct 30 04:45:05.421658 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.421644    5072 scope.go:122] "RemoveContainer" containerID="5174075df0ef240cddde365a9885a3889f4649bb1b1659de3b2395804a2a2e75"
Oct 30 04:45:05.421926 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:05.421903    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86_odh-ai-gateway-infra(25763423-975c-49f4-88a9-10b9e38015ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86" podUID="25763423-975c-49f4-88a9-10b9e38015ea"
Oct 30 04:45:05.423278 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.423253    5072 generic.go:356] "Generic (PLEG): container finished" podID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" containerID="1a98182e71eade579f14e19fbfc0cc134b767e2dc86d00f97e6d530e80f658cd" exitCode=60
Oct 30 04:45:05.423369 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.423336    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5" event={"ID":"5a984c77-fa0d-4970-a1a3-55351ab6e7ee","Type":"ContainerDied","Data":"1a98182e71eade579f14e19fbfc0cc134b767e2dc86d00f97e6d530e80f658cd"}
Oct 30 04:45:05.423557 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.423539    5072 scope.go:122] "RemoveContainer" containerID="1a98182e71eade579f14e19fbfc0cc134b767e2dc86d00f97e6d530e80f658cd"
Oct 30 04:45:05.423827 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:05.423797    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888925-m7sg5_odh-ai-gateway-infra(5a984c77-fa0d-4970-a1a3-55351ab6e7ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee"
Oct 30 04:45:05.425331 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.425311    5072 generic.go:356] "Generic (PLEG): container finished" podID="a7d8b785-0448-4ed4-b17a-38d43794b893" containerID="bd889596a85c0e95d602798085895f92275c8d1801a88fe2d74924883bf6df81" exitCode=60
Oct 30 04:45:05.425436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.425380    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2" event={"ID":"a7d8b785-0448-4ed4-b17a-38d43794b893","Type":"ContainerDied","Data":"bd889596a85c0e95d602798085895f92275c8d1801a88fe2d74924883bf6df81"}
Oct 30 04:45:05.425638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.425618    5072 scope.go:122] "RemoveContainer" containerID="bd889596a85c0e95d602798085895f92275c8d1801a88fe2d74924883bf6df81"
Oct 30 04:45:05.425922 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:05.425896    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2_odh-ai-gateway-infra(a7d8b785-0448-4ed4-b17a-38d43794b893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893"
Oct 30 04:45:05.427265 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.427248    5072 generic.go:356] "Generic (PLEG): container finished" podID="3abc0c26-3d30-4303-826f-7af0b072fc56" containerID="7b82bfddaca0b6d5aa2c6b6f03be9f990329c2578801b3b7c4203d2759711d9c" exitCode=60
Oct 30 04:45:05.427352 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.427326    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx" event={"ID":"3abc0c26-3d30-4303-826f-7af0b072fc56","Type":"ContainerDied","Data":"7b82bfddaca0b6d5aa2c6b6f03be9f990329c2578801b3b7c4203d2759711d9c"}
Oct 30 04:45:05.427524 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.427510    5072 scope.go:122] "RemoveContainer" containerID="7b82bfddaca0b6d5aa2c6b6f03be9f990329c2578801b3b7c4203d2759711d9c"
Oct 30 04:45:05.427727 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:05.427709    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx_odh-ai-gateway-infra(3abc0c26-3d30-4303-826f-7af0b072fc56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56"
Oct 30 04:45:05.429180 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.429159    5072 generic.go:356] "Generic (PLEG): container finished" podID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" containerID="537be6e231df499178ed06407302a00f98910f7fc39a1b1ec1bc90fef4391850" exitCode=60
Oct 30 04:45:05.429280 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.429216    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt" event={"ID":"2da7ba2d-6b50-4f01-8dc2-38552e35fd80","Type":"ContainerDied","Data":"537be6e231df499178ed06407302a00f98910f7fc39a1b1ec1bc90fef4391850"}
Oct 30 04:45:05.429378 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.429364    5072 scope.go:122] "RemoveContainer" containerID="537be6e231df499178ed06407302a00f98910f7fc39a1b1ec1bc90fef4391850"
Oct 30 04:45:05.429571 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:05.429551    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt_odh-ai-gateway-infra(2da7ba2d-6b50-4f01-8dc2-38552e35fd80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80"
Oct 30 04:45:05.430986 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.430967    5072 scope.go:122] "RemoveContainer" containerID="4ab4a8708611fc9945cdae4b8e6f171123393f7333a98f143f049c5e55061ae3"
Oct 30 04:45:05.431206 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:05.431178    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg_odh-ai-gateway-infra(52403e2b-a240-452e-98f1-c577b1f770db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg" podUID="52403e2b-a240-452e-98f1-c577b1f770db"
Oct 30 04:45:05.432785 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.432765    5072 generic.go:356] "Generic (PLEG): container finished" podID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" containerID="d9d726a04cdeff0524bc23af60351de8ef1b9deba9d71bddc3260abac9a2aacb" exitCode=60
Oct 30 04:45:05.432880 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.432816    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824" event={"ID":"a7aa09dd-6974-44f5-9cc3-cbaba2223946","Type":"ContainerDied","Data":"d9d726a04cdeff0524bc23af60351de8ef1b9deba9d71bddc3260abac9a2aacb"}
Oct 30 04:45:05.433053 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.433037    5072 scope.go:122] "RemoveContainer" containerID="d9d726a04cdeff0524bc23af60351de8ef1b9deba9d71bddc3260abac9a2aacb"
Oct 30 04:45:05.434774 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:05.434750    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824_odh-ai-gateway-infra(a7aa09dd-6974-44f5-9cc3-cbaba2223946)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946"
Oct 30 04:45:05.453308 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.453282    5072 scope.go:122] "RemoveContainer" containerID="61e4202e68d02402d22f301513a0c9c0bed4a2a66c4e2de8ef1193ed9b0fed73"
Oct 30 04:45:05.469065 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.468950    5072 scope.go:122] "RemoveContainer" containerID="d67951118cd63b2aa33fabbbe4a7a421c21c8c058f2cef65168664024989eca0"
Oct 30 04:45:05.494789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.494764    5072 scope.go:122] "RemoveContainer" containerID="61b0bda0ce70d1beaa136ae728595a4d0a51942ea171b0ebe9bbf6998ed6e7f1"
Oct 30 04:45:05.515279 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.515243    5072 scope.go:122] "RemoveContainer" containerID="dc73ea2158dc4a3bc99d38899cca578a0f3d8349ea5ec1fba8230e7898151743"
Oct 30 04:45:05.536308 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.536280    5072 scope.go:122] "RemoveContainer" containerID="b0c7ee16b4b94e43832b587fed65ec211c28a72421e5051996c4c793f3900a1e"
Oct 30 04:45:05.555634 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.555607    5072 scope.go:122] "RemoveContainer" containerID="95e3133ecf0e8c1d3bcba6605c0e88639436e72df6e4e55906a139923f4355a5"
Oct 30 04:45:05.575965 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.575602    5072 scope.go:122] "RemoveContainer" containerID="0bb885803bbaedc8077886d5bc68dd693c7eaf40e65f9e425d950297fcb02c98"
Oct 30 04:45:05.659647 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.659614    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:45:05.683591 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.683551    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe067687-49f8-4781-9c02-a2b47ea35293-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe067687-49f8-4781-9c02-a2b47ea35293-tls-certs\") pod \"fe067687-49f8-4781-9c02-a2b47ea35293\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") "
Oct 30 04:45:05.683827 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.683758    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-dshm\") pod \"fe067687-49f8-4781-9c02-a2b47ea35293\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") "
Oct 30 04:45:05.683827 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.683810    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe067687-49f8-4781-9c02-a2b47ea35293-kube-api-access-kl92n\" (UniqueName: \"kubernetes.io/projected/fe067687-49f8-4781-9c02-a2b47ea35293-kube-api-access-kl92n\") pod \"fe067687-49f8-4781-9c02-a2b47ea35293\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") "
Oct 30 04:45:05.683947 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.683917    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-home\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-home\") pod \"fe067687-49f8-4781-9c02-a2b47ea35293\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") "
Oct 30 04:45:05.683947 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.683941    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-tmp-dir\") pod \"fe067687-49f8-4781-9c02-a2b47ea35293\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") "
Oct 30 04:45:05.684206 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.684182    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-tmp-dir" pod "fe067687-49f8-4781-9c02-a2b47ea35293" (UID: "fe067687-49f8-4781-9c02-a2b47ea35293"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:05.684313 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.684259    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-model-cache\") pod \"fe067687-49f8-4781-9c02-a2b47ea35293\" (UID: \"fe067687-49f8-4781-9c02-a2b47ea35293\") "
Oct 30 04:45:05.684567 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.684522    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-home" pod "fe067687-49f8-4781-9c02-a2b47ea35293" (UID: "fe067687-49f8-4781-9c02-a2b47ea35293"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:05.684730 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.684597    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-model-cache" pod "fe067687-49f8-4781-9c02-a2b47ea35293" (UID: "fe067687-49f8-4781-9c02-a2b47ea35293"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:05.684992 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.684965    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:05.685129 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.684997    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:05.685129 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.685011    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:05.686312 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.686291    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe067687-49f8-4781-9c02-a2b47ea35293-tls-certs" pod "fe067687-49f8-4781-9c02-a2b47ea35293" (UID: "fe067687-49f8-4781-9c02-a2b47ea35293"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:45:05.686674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.686656    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe067687-49f8-4781-9c02-a2b47ea35293-kube-api-access-kl92n" pod "fe067687-49f8-4781-9c02-a2b47ea35293" (UID: "fe067687-49f8-4781-9c02-a2b47ea35293"). InnerVolumeSpecName "kube-api-access-kl92n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:05.686674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.686656    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-dshm" pod "fe067687-49f8-4781-9c02-a2b47ea35293" (UID: "fe067687-49f8-4781-9c02-a2b47ea35293"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:05.789150 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.789099    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe067687-49f8-4781-9c02-a2b47ea35293-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:05.789150 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.789139    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl92n\" (UniqueName: \"kubernetes.io/projected/fe067687-49f8-4781-9c02-a2b47ea35293-kube-api-access-kl92n\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:05.789150 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:05.789154    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe067687-49f8-4781-9c02-a2b47ea35293-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:06.442933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.442905    5072 scope.go:122] "RemoveContainer" containerID="bd889596a85c0e95d602798085895f92275c8d1801a88fe2d74924883bf6df81"
Oct 30 04:45:06.443382 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:06.443244    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2_odh-ai-gateway-infra(a7d8b785-0448-4ed4-b17a-38d43794b893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893"
Oct 30 04:45:06.444658 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.444637    5072 scope.go:122] "RemoveContainer" containerID="7b82bfddaca0b6d5aa2c6b6f03be9f990329c2578801b3b7c4203d2759711d9c"
Oct 30 04:45:06.444983 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:06.444958    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx_odh-ai-gateway-infra(3abc0c26-3d30-4303-826f-7af0b072fc56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56"
Oct 30 04:45:06.446251 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.446234    5072 scope.go:122] "RemoveContainer" containerID="537be6e231df499178ed06407302a00f98910f7fc39a1b1ec1bc90fef4391850"
Oct 30 04:45:06.446499 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:06.446477    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt_odh-ai-gateway-infra(2da7ba2d-6b50-4f01-8dc2-38552e35fd80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80"
Oct 30 04:45:06.447856 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.447835    5072 scope.go:122] "RemoveContainer" containerID="d9d726a04cdeff0524bc23af60351de8ef1b9deba9d71bddc3260abac9a2aacb"
Oct 30 04:45:06.448138 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:06.448117    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824_odh-ai-gateway-infra(a7aa09dd-6974-44f5-9cc3-cbaba2223946)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946"
Oct 30 04:45:06.449399 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.449384    5072 scope.go:122] "RemoveContainer" containerID="438f102abbf5b96371977baa58368ca94d546b32689b2b0173f75bd7f364172c"
Oct 30 04:45:06.449596 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:06.449573    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv_odh-ai-gateway-infra(eac636d9-a1c1-4220-974f-46a2666f0005)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv" podUID="eac636d9-a1c1-4220-974f-46a2666f0005"
Oct 30 04:45:06.450486 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.450465    5072 generic.go:356] "Generic (PLEG): container finished" podID="fe067687-49f8-4781-9c02-a2b47ea35293" containerID="3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334" exitCode=0
Oct 30 04:45:06.450586 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.450496    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"
Oct 30 04:45:06.450586 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.450539    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26" event={"ID":"fe067687-49f8-4781-9c02-a2b47ea35293","Type":"ContainerDied","Data":"3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334"}
Oct 30 04:45:06.450586 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.450577    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26" event={"ID":"fe067687-49f8-4781-9c02-a2b47ea35293","Type":"ContainerDied","Data":"a8bd9a9c61c1fce8da136fd8ccbeb570d109d658d5d49d7554d7ea0fa7c15117"}
Oct 30 04:45:06.450735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.450595    5072 scope.go:122] "RemoveContainer" containerID="3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334"
Oct 30 04:45:06.452859 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.452825    5072 scope.go:122] "RemoveContainer" containerID="b3a512fa5607e8d6526aecf62dbcfe5481ed0c9d7e5c050b42e7d3c912a4f019"
Oct 30 04:45:06.453050 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:06.453031    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx_odh-ai-gateway-infra(900d8674-41c1-434e-86cf-562bf6ef0822)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx" podUID="900d8674-41c1-434e-86cf-562bf6ef0822"
Oct 30 04:45:06.454613 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.454598    5072 scope.go:122] "RemoveContainer" containerID="5174075df0ef240cddde365a9885a3889f4649bb1b1659de3b2395804a2a2e75"
Oct 30 04:45:06.454802 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:06.454783    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86_odh-ai-gateway-infra(25763423-975c-49f4-88a9-10b9e38015ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86" podUID="25763423-975c-49f4-88a9-10b9e38015ea"
Oct 30 04:45:06.456469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.456451    5072 scope.go:122] "RemoveContainer" containerID="1a98182e71eade579f14e19fbfc0cc134b767e2dc86d00f97e6d530e80f658cd"
Oct 30 04:45:06.456692 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:06.456672    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888925-m7sg5_odh-ai-gateway-infra(5a984c77-fa0d-4970-a1a3-55351ab6e7ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee"
Oct 30 04:45:06.462250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.462229    5072 scope.go:122] "RemoveContainer" containerID="3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334"
Oct 30 04:45:06.462531 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:06.462510    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334\": container with ID starting with 3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334 not found: ID does not exist" containerID="3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334"
Oct 30 04:45:06.462571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.462538    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334"} err="failed to get container status \"3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334\": rpc error: code = NotFound desc = could not find container \"3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334\": container with ID starting with 3d2ff29a08a1e92b9df5dd149282597de2eb4c413dc4a2196c85d95476eb5334 not found: ID does not exist"
Oct 30 04:45:06.602790 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.602740    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26" podUID="fe067687-49f8-4781-9c02-a2b47ea35293" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:45:06.701785 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.701755    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"]
Oct 30 04:45:06.709860 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:06.709821    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e070ea/auth-test-model-e070ea-kserve-7b84868f5c-9wx26"]
Oct 30 04:45:07.259525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.259493    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"]
Oct 30 04:45:07.260236 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.260218    5072 cpu_manager.go:414] "containerMap: removing container" podUID="fe067687-49f8-4781-9c02-a2b47ea35293" containerName="main"
Oct 30 04:45:07.260236 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.260235    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe067687-49f8-4781-9c02-a2b47ea35293" containerName="main"
Oct 30 04:45:07.260377 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.260356    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe067687-49f8-4781-9c02-a2b47ea35293" containerName="main"
Oct 30 04:45:07.270817 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.270784    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"
Oct 30 04:45:07.310417 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.310377    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c0e964\""
Oct 30 04:45:07.310417 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.310411    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c0e964-dockercfg-w4cqf\""
Oct 30 04:45:07.315058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.314178    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh\" (UID: \"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"
Oct 30 04:45:07.315058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.314367    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx4zd\" (UniqueName: \"kubernetes.io/projected/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-kube-api-access-wx4zd\") pod \"payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh\" (UID: \"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"
Oct 30 04:45:07.319363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.319341    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"]
Oct 30 04:45:07.323642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.323617    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"]
Oct 30 04:45:07.328486 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.328460    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"
Oct 30 04:45:07.349520 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.349491    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"]
Oct 30 04:45:07.426351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.426318    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh\" (UID: \"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"
Oct 30 04:45:07.426539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.426410    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx4zd\" (UniqueName: \"kubernetes.io/projected/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-kube-api-access-wx4zd\") pod \"payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh\" (UID: \"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"
Oct 30 04:45:07.426539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.426436    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc\" (UID: \"93480c6a-fe04-426b-9ee4-cdfd827f2ab9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"
Oct 30 04:45:07.426539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.426485    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlwrh\" (UniqueName: \"kubernetes.io/projected/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-kube-api-access-zlwrh\") pod \"payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc\" (UID: \"93480c6a-fe04-426b-9ee4-cdfd827f2ab9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"
Oct 30 04:45:07.428455 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.428417    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh\" (UID: \"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"
Oct 30 04:45:07.464623 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.464593    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx4zd\" (UniqueName: \"kubernetes.io/projected/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-kube-api-access-wx4zd\") pod \"payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh\" (UID: \"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"
Oct 30 04:45:07.527780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.527686    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlwrh\" (UniqueName: \"kubernetes.io/projected/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-kube-api-access-zlwrh\") pod \"payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc\" (UID: \"93480c6a-fe04-426b-9ee4-cdfd827f2ab9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"
Oct 30 04:45:07.527948 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.527874    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc\" (UID: \"93480c6a-fe04-426b-9ee4-cdfd827f2ab9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"
Oct 30 04:45:07.528741 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.528717    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc\" (UID: \"93480c6a-fe04-426b-9ee4-cdfd827f2ab9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"
Oct 30 04:45:07.570493 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.570455    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlwrh\" (UniqueName: \"kubernetes.io/projected/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-kube-api-access-zlwrh\") pod \"payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc\" (UID: \"93480c6a-fe04-426b-9ee4-cdfd827f2ab9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"
Oct 30 04:45:07.590533 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.590498    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"
Oct 30 04:45:07.644879 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.644845    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"
Oct 30 04:45:07.901462 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:45:07.901431    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec101f2_4aa8_4a85_ab2b_a30f9fe298f0.slice/crio-ea32d689bda567064bbd7474ead066d161b6976ae095df048602852188db07e5 WatchSource:0}: Error finding container ea32d689bda567064bbd7474ead066d161b6976ae095df048602852188db07e5: Status 404 returned error can't find the container with id ea32d689bda567064bbd7474ead066d161b6976ae095df048602852188db07e5
Oct 30 04:45:07.902167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.902145    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"]
Oct 30 04:45:07.967937 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:45:07.967903    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93480c6a_fe04_426b_9ee4_cdfd827f2ab9.slice/crio-fc86e527c6f1b0c32f98ae8969e44a19ec72705b09ef843b494c70dfa924144f WatchSource:0}: Error finding container fc86e527c6f1b0c32f98ae8969e44a19ec72705b09ef843b494c70dfa924144f: Status 404 returned error can't find the container with id fc86e527c6f1b0c32f98ae8969e44a19ec72705b09ef843b494c70dfa924144f
Oct 30 04:45:07.968348 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:07.968326    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"]
Oct 30 04:45:08.410491 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:08.410453    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe067687-49f8-4781-9c02-a2b47ea35293" path="/var/lib/kubelet/pods/fe067687-49f8-4781-9c02-a2b47ea35293/volumes"
Oct 30 04:45:08.474444 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:08.474402    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh" event={"ID":"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0","Type":"ContainerStarted","Data":"10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254"}
Oct 30 04:45:08.474930 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:08.474474    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh" event={"ID":"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0","Type":"ContainerStarted","Data":"ea32d689bda567064bbd7474ead066d161b6976ae095df048602852188db07e5"}
Oct 30 04:45:08.474930 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:08.474622    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"
Oct 30 04:45:08.476296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:08.476266    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc" event={"ID":"93480c6a-fe04-426b-9ee4-cdfd827f2ab9","Type":"ContainerStarted","Data":"5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b"}
Oct 30 04:45:08.476423 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:08.476319    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc" event={"ID":"93480c6a-fe04-426b-9ee4-cdfd827f2ab9","Type":"ContainerStarted","Data":"fc86e527c6f1b0c32f98ae8969e44a19ec72705b09ef843b494c70dfa924144f"}
Oct 30 04:45:08.476423 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:08.476344    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"
Oct 30 04:45:08.518723 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:08.518671    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh" podStartSLOduration=1.5186592559999998 podStartE2EDuration="1.518659256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:08.515605211 +0000 UTC m=+2888.257028309" watchObservedRunningTime="2026-10-30 04:45:08.518659256 +0000 UTC m=+2888.260082352"
Oct 30 04:45:08.561813 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:08.561751    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc" podStartSLOduration=1.5617336659999999 podStartE2EDuration="1.561733666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:08.557994246 +0000 UTC m=+2888.299417342" watchObservedRunningTime="2026-10-30 04:45:08.561733666 +0000 UTC m=+2888.303156767"
Oct 30 04:45:10.252164 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:10.252138    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"
Oct 30 04:45:10.252579 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:10.252205    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"
Oct 30 04:45:11.278421 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:11.278386    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"
Oct 30 04:45:11.278850 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:11.278467    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"
Oct 30 04:45:11.754256 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:11.754217    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:45:14.400598 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:14.400566    5072 scope.go:122] "RemoveContainer" containerID="4ab4a8708611fc9945cdae4b8e6f171123393f7333a98f143f049c5e55061ae3"
Oct 30 04:45:14.610150 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:14.610106    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk" podUID="8b1ad519-9858-4903-847c-e8dc14645411" containerName="main" containerID="cri-o://ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b" gracePeriod=45
Oct 30 04:45:14.899677 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:14.899647    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:45:15.020912 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.020822    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-tmp-dir\") pod \"8b1ad519-9858-4903-847c-e8dc14645411\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") "
Oct 30 04:45:15.020912 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.020875    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-dshm\") pod \"8b1ad519-9858-4903-847c-e8dc14645411\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") "
Oct 30 04:45:15.020912 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.020895    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1ad519-9858-4903-847c-e8dc14645411-kube-api-access-44lcz\" (UniqueName: \"kubernetes.io/projected/8b1ad519-9858-4903-847c-e8dc14645411-kube-api-access-44lcz\") pod \"8b1ad519-9858-4903-847c-e8dc14645411\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") "
Oct 30 04:45:15.021263 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.020919    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-home\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-home\") pod \"8b1ad519-9858-4903-847c-e8dc14645411\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") "
Oct 30 04:45:15.021263 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.021000    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b1ad519-9858-4903-847c-e8dc14645411-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1ad519-9858-4903-847c-e8dc14645411-tls-certs\") pod \"8b1ad519-9858-4903-847c-e8dc14645411\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") "
Oct 30 04:45:15.021263 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.021143    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-model-cache\") pod \"8b1ad519-9858-4903-847c-e8dc14645411\" (UID: \"8b1ad519-9858-4903-847c-e8dc14645411\") "
Oct 30 04:45:15.021263 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.021197    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-tmp-dir" pod "8b1ad519-9858-4903-847c-e8dc14645411" (UID: "8b1ad519-9858-4903-847c-e8dc14645411"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:15.021469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.021313    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-home" pod "8b1ad519-9858-4903-847c-e8dc14645411" (UID: "8b1ad519-9858-4903-847c-e8dc14645411"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:15.021469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.021459    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-model-cache" pod "8b1ad519-9858-4903-847c-e8dc14645411" (UID: "8b1ad519-9858-4903-847c-e8dc14645411"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:15.021803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.021735    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:15.021803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.021753    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:15.021803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.021762    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:15.023475 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.023452    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-dshm" pod "8b1ad519-9858-4903-847c-e8dc14645411" (UID: "8b1ad519-9858-4903-847c-e8dc14645411"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:15.023566 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.023497    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1ad519-9858-4903-847c-e8dc14645411-kube-api-access-44lcz" pod "8b1ad519-9858-4903-847c-e8dc14645411" (UID: "8b1ad519-9858-4903-847c-e8dc14645411"). InnerVolumeSpecName "kube-api-access-44lcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:15.023566 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.023503    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b1ad519-9858-4903-847c-e8dc14645411-tls-certs" pod "8b1ad519-9858-4903-847c-e8dc14645411" (UID: "8b1ad519-9858-4903-847c-e8dc14645411"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:45:15.122521 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.122487    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1ad519-9858-4903-847c-e8dc14645411-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:15.122521 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.122517    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b1ad519-9858-4903-847c-e8dc14645411-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:15.122521 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.122526    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44lcz\" (UniqueName: \"kubernetes.io/projected/8b1ad519-9858-4903-847c-e8dc14645411-kube-api-access-44lcz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:15.436267 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.436243    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"
Oct 30 04:45:15.436672 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.436655    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"
Oct 30 04:45:15.531346 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.531314    5072 generic.go:356] "Generic (PLEG): container finished" podID="8b1ad519-9858-4903-847c-e8dc14645411" containerID="ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b" exitCode=0
Oct 30 04:45:15.531555 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.531379    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"
Oct 30 04:45:15.531555 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.531387    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk" event={"ID":"8b1ad519-9858-4903-847c-e8dc14645411","Type":"ContainerDied","Data":"ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b"}
Oct 30 04:45:15.531555 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.531434    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk" event={"ID":"8b1ad519-9858-4903-847c-e8dc14645411","Type":"ContainerDied","Data":"848519b6e7606b62b2b14c51e0b96c36ceb21a2179c79f00882cb95b9dbb50ee"}
Oct 30 04:45:15.531555 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.531478    5072 scope.go:122] "RemoveContainer" containerID="ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b"
Oct 30 04:45:15.534376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.534350    5072 generic.go:356] "Generic (PLEG): container finished" podID="52403e2b-a240-452e-98f1-c577b1f770db" containerID="6b2d5d6b9abd6b4fe918eb80b4ac2238ed44e98dccb3698a82ae4ff3aa5c3d81" exitCode=60
Oct 30 04:45:15.534510 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.534428    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg" event={"ID":"52403e2b-a240-452e-98f1-c577b1f770db","Type":"ContainerDied","Data":"6b2d5d6b9abd6b4fe918eb80b4ac2238ed44e98dccb3698a82ae4ff3aa5c3d81"}
Oct 30 04:45:15.534868 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.534852    5072 scope.go:122] "RemoveContainer" containerID="6b2d5d6b9abd6b4fe918eb80b4ac2238ed44e98dccb3698a82ae4ff3aa5c3d81"
Oct 30 04:45:15.535230 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:15.535205    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg_odh-ai-gateway-infra(52403e2b-a240-452e-98f1-c577b1f770db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg" podUID="52403e2b-a240-452e-98f1-c577b1f770db"
Oct 30 04:45:15.544394 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.544308    5072 scope.go:122] "RemoveContainer" containerID="ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b"
Oct 30 04:45:15.544805 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:15.544776    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b\": container with ID starting with ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b not found: ID does not exist" containerID="ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b"
Oct 30 04:45:15.544889 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.544815    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b"} err="failed to get container status \"ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b\": rpc error: code = NotFound desc = could not find container \"ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b\": container with ID starting with ece031c46fcb86690da1a9aeb3976345611e1cb2ae3faefceb02e06363f07e1b not found: ID does not exist"
Oct 30 04:45:15.544889 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.544841    5072 scope.go:122] "RemoveContainer" containerID="4ab4a8708611fc9945cdae4b8e6f171123393f7333a98f143f049c5e55061ae3"
Oct 30 04:45:15.668487 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.668456    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"]
Oct 30 04:45:15.688111 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.688069    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk"]
Oct 30 04:45:15.889649 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:15.889600    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-247158/auth-test-model-247158-kserve-78bc8ff669-hq2dk" podUID="8b1ad519-9858-4903-847c-e8dc14645411" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:45:16.413430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:16.409213    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1ad519-9858-4903-847c-e8dc14645411" path="/var/lib/kubelet/pods/8b1ad519-9858-4903-847c-e8dc14645411/volumes"
Oct 30 04:45:16.757320 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:16.757278    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"]
Oct 30 04:45:16.908750 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:16.908722    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"
Oct 30 04:45:17.044835 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.044737    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52403e2b-a240-452e-98f1-c577b1f770db-kube-api-access-rgscn\" (UniqueName: \"kubernetes.io/projected/52403e2b-a240-452e-98f1-c577b1f770db-kube-api-access-rgscn\") pod \"52403e2b-a240-452e-98f1-c577b1f770db\" (UID: \"52403e2b-a240-452e-98f1-c577b1f770db\") "
Oct 30 04:45:17.045011 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.044840    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52403e2b-a240-452e-98f1-c577b1f770db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52403e2b-a240-452e-98f1-c577b1f770db-maas-api-service-ca\") pod \"52403e2b-a240-452e-98f1-c577b1f770db\" (UID: \"52403e2b-a240-452e-98f1-c577b1f770db\") "
Oct 30 04:45:17.045290 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.045263    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52403e2b-a240-452e-98f1-c577b1f770db-maas-api-service-ca" pod "52403e2b-a240-452e-98f1-c577b1f770db" (UID: "52403e2b-a240-452e-98f1-c577b1f770db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:17.047065 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.047042    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52403e2b-a240-452e-98f1-c577b1f770db-kube-api-access-rgscn" pod "52403e2b-a240-452e-98f1-c577b1f770db" (UID: "52403e2b-a240-452e-98f1-c577b1f770db"). InnerVolumeSpecName "kube-api-access-rgscn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:17.146610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.146580    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgscn\" (UniqueName: \"kubernetes.io/projected/52403e2b-a240-452e-98f1-c577b1f770db-kube-api-access-rgscn\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:17.146610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.146607    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52403e2b-a240-452e-98f1-c577b1f770db-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:17.401500 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.401406    5072 scope.go:122] "RemoveContainer" containerID="5174075df0ef240cddde365a9885a3889f4649bb1b1659de3b2395804a2a2e75"
Oct 30 04:45:17.401653 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.401508    5072 scope.go:122] "RemoveContainer" containerID="438f102abbf5b96371977baa58368ca94d546b32689b2b0173f75bd7f364172c"
Oct 30 04:45:17.401653 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.401551    5072 scope.go:122] "RemoveContainer" containerID="d9d726a04cdeff0524bc23af60351de8ef1b9deba9d71bddc3260abac9a2aacb"
Oct 30 04:45:17.401653 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.401626    5072 scope.go:122] "RemoveContainer" containerID="7b82bfddaca0b6d5aa2c6b6f03be9f990329c2578801b3b7c4203d2759711d9c"
Oct 30 04:45:17.401863 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.401843    5072 scope.go:122] "RemoveContainer" containerID="bd889596a85c0e95d602798085895f92275c8d1801a88fe2d74924883bf6df81"
Oct 30 04:45:17.551597 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.551571    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"
Oct 30 04:45:17.551756 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.551607    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg" event={"ID":"52403e2b-a240-452e-98f1-c577b1f770db","Type":"ContainerDied","Data":"5428020108eed706e392d9aba5e2f6ab313ef78f79f12338486a6d8102ae3132"}
Oct 30 04:45:17.551756 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.551668    5072 scope.go:122] "RemoveContainer" containerID="6b2d5d6b9abd6b4fe918eb80b4ac2238ed44e98dccb3698a82ae4ff3aa5c3d81"
Oct 30 04:45:17.609578 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.609556    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"]
Oct 30 04:45:17.616755 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:17.616731    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888925-6dmdg"]
Oct 30 04:45:18.401116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.401061    5072 scope.go:122] "RemoveContainer" containerID="b3a512fa5607e8d6526aecf62dbcfe5481ed0c9d7e5c050b42e7d3c912a4f019"
Oct 30 04:45:18.401518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.401377    5072 scope.go:122] "RemoveContainer" containerID="537be6e231df499178ed06407302a00f98910f7fc39a1b1ec1bc90fef4391850"
Oct 30 04:45:18.410148 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.410050    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52403e2b-a240-452e-98f1-c577b1f770db" path="/var/lib/kubelet/pods/52403e2b-a240-452e-98f1-c577b1f770db/volumes"
Oct 30 04:45:18.562436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.562408    5072 generic.go:356] "Generic (PLEG): container finished" podID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" containerID="aa81363115971dacf90a388913333ed01eac773299b6592c6654695a61a7795f" exitCode=60
Oct 30 04:45:18.562587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.562475    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824" event={"ID":"a7aa09dd-6974-44f5-9cc3-cbaba2223946","Type":"ContainerDied","Data":"aa81363115971dacf90a388913333ed01eac773299b6592c6654695a61a7795f"}
Oct 30 04:45:18.562587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.562533    5072 scope.go:122] "RemoveContainer" containerID="d9d726a04cdeff0524bc23af60351de8ef1b9deba9d71bddc3260abac9a2aacb"
Oct 30 04:45:18.562942 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.562923    5072 scope.go:122] "RemoveContainer" containerID="aa81363115971dacf90a388913333ed01eac773299b6592c6654695a61a7795f"
Oct 30 04:45:18.563287 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:18.563260    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824_odh-ai-gateway-infra(a7aa09dd-6974-44f5-9cc3-cbaba2223946)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946"
Oct 30 04:45:18.565120 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.565096    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv" event={"ID":"eac636d9-a1c1-4220-974f-46a2666f0005","Type":"ContainerStarted","Data":"496a93637872c60c975870241c30473e5eb0ddd1225c0180a35563944d884921"}
Oct 30 04:45:18.568165 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.568145    5072 generic.go:356] "Generic (PLEG): container finished" podID="25763423-975c-49f4-88a9-10b9e38015ea" containerID="aa393d87dab07093532c654ae84c94b41bdfad90b03442f901f115003076fe95" exitCode=60
Oct 30 04:45:18.568250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.568208    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86" event={"ID":"25763423-975c-49f4-88a9-10b9e38015ea","Type":"ContainerDied","Data":"aa393d87dab07093532c654ae84c94b41bdfad90b03442f901f115003076fe95"}
Oct 30 04:45:18.568542 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.568527    5072 scope.go:122] "RemoveContainer" containerID="aa393d87dab07093532c654ae84c94b41bdfad90b03442f901f115003076fe95"
Oct 30 04:45:18.568782 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:18.568763    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86_odh-ai-gateway-infra(25763423-975c-49f4-88a9-10b9e38015ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86" podUID="25763423-975c-49f4-88a9-10b9e38015ea"
Oct 30 04:45:18.570442 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.570418    5072 generic.go:356] "Generic (PLEG): container finished" podID="a7d8b785-0448-4ed4-b17a-38d43794b893" containerID="9d193908d031f2264ad7239a8f6d2ac2371ab55a121aba2f895dc1f1c8a0892f" exitCode=60
Oct 30 04:45:18.570523 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.570482    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2" event={"ID":"a7d8b785-0448-4ed4-b17a-38d43794b893","Type":"ContainerDied","Data":"9d193908d031f2264ad7239a8f6d2ac2371ab55a121aba2f895dc1f1c8a0892f"}
Oct 30 04:45:18.570830 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.570812    5072 scope.go:122] "RemoveContainer" containerID="9d193908d031f2264ad7239a8f6d2ac2371ab55a121aba2f895dc1f1c8a0892f"
Oct 30 04:45:18.571104 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:18.571058    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2_odh-ai-gateway-infra(a7d8b785-0448-4ed4-b17a-38d43794b893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893"
Oct 30 04:45:18.572576 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.572554    5072 generic.go:356] "Generic (PLEG): container finished" podID="3abc0c26-3d30-4303-826f-7af0b072fc56" containerID="32b19f1065e98d9bcd8324041549a8ae24af13f919704a5913dfb86aa1378238" exitCode=60
Oct 30 04:45:18.572657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.572606    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx" event={"ID":"3abc0c26-3d30-4303-826f-7af0b072fc56","Type":"ContainerDied","Data":"32b19f1065e98d9bcd8324041549a8ae24af13f919704a5913dfb86aa1378238"}
Oct 30 04:45:18.572875 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.572857    5072 scope.go:122] "RemoveContainer" containerID="32b19f1065e98d9bcd8324041549a8ae24af13f919704a5913dfb86aa1378238"
Oct 30 04:45:18.573046 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:18.573030    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx_odh-ai-gateway-infra(3abc0c26-3d30-4303-826f-7af0b072fc56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56"
Oct 30 04:45:18.626649 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.626626    5072 scope.go:122] "RemoveContainer" containerID="5174075df0ef240cddde365a9885a3889f4649bb1b1659de3b2395804a2a2e75"
Oct 30 04:45:18.648060 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.648039    5072 scope.go:122] "RemoveContainer" containerID="bd889596a85c0e95d602798085895f92275c8d1801a88fe2d74924883bf6df81"
Oct 30 04:45:18.721605 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:18.721583    5072 scope.go:122] "RemoveContainer" containerID="7b82bfddaca0b6d5aa2c6b6f03be9f990329c2578801b3b7c4203d2759711d9c"
Oct 30 04:45:19.401218 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.401189    5072 scope.go:122] "RemoveContainer" containerID="1a98182e71eade579f14e19fbfc0cc134b767e2dc86d00f97e6d530e80f658cd"
Oct 30 04:45:19.486992 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.486971    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"
Oct 30 04:45:19.487120 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.487039    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"
Oct 30 04:45:19.584708 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.584662    5072 generic.go:356] "Generic (PLEG): container finished" podID="eac636d9-a1c1-4220-974f-46a2666f0005" containerID="496a93637872c60c975870241c30473e5eb0ddd1225c0180a35563944d884921" exitCode=60
Oct 30 04:45:19.584887 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.584726    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv" event={"ID":"eac636d9-a1c1-4220-974f-46a2666f0005","Type":"ContainerDied","Data":"496a93637872c60c975870241c30473e5eb0ddd1225c0180a35563944d884921"}
Oct 30 04:45:19.584887 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.584787    5072 scope.go:122] "RemoveContainer" containerID="438f102abbf5b96371977baa58368ca94d546b32689b2b0173f75bd7f364172c"
Oct 30 04:45:19.585209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.585191    5072 scope.go:122] "RemoveContainer" containerID="496a93637872c60c975870241c30473e5eb0ddd1225c0180a35563944d884921"
Oct 30 04:45:19.585506 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:19.585485    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv_odh-ai-gateway-infra(eac636d9-a1c1-4220-974f-46a2666f0005)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv" podUID="eac636d9-a1c1-4220-974f-46a2666f0005"
Oct 30 04:45:19.594875 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.590976    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx" event={"ID":"900d8674-41c1-434e-86cf-562bf6ef0822","Type":"ContainerStarted","Data":"4684b9cf730cf7fc9e3c4120b459ee6c3ec5a750b4688480dd3839b8e45ec7fa"}
Oct 30 04:45:19.600531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.600497    5072 generic.go:356] "Generic (PLEG): container finished" podID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" containerID="1a9c49c0b17cba8c826be5221cc569cc610b4619e8e1d15b6e76ebf9c3e852a0" exitCode=60
Oct 30 04:45:19.600717 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.600634    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt" event={"ID":"2da7ba2d-6b50-4f01-8dc2-38552e35fd80","Type":"ContainerDied","Data":"1a9c49c0b17cba8c826be5221cc569cc610b4619e8e1d15b6e76ebf9c3e852a0"}
Oct 30 04:45:19.601242 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.601066    5072 scope.go:122] "RemoveContainer" containerID="1a9c49c0b17cba8c826be5221cc569cc610b4619e8e1d15b6e76ebf9c3e852a0"
Oct 30 04:45:19.601446 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:19.601419    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt_odh-ai-gateway-infra(2da7ba2d-6b50-4f01-8dc2-38552e35fd80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80"
Oct 30 04:45:19.607430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.607273    5072 scope.go:122] "RemoveContainer" containerID="537be6e231df499178ed06407302a00f98910f7fc39a1b1ec1bc90fef4391850"
Oct 30 04:45:19.614793 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.614760    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"]
Oct 30 04:45:19.669987 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.669951    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"]
Oct 30 04:45:19.719681 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.719630    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"]
Oct 30 04:45:19.747928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.747759    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"]
Oct 30 04:45:19.802343 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.802207    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"]
Oct 30 04:45:19.840417 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.840334    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"
Oct 30 04:45:19.880117 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.879985    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7aa09dd-6974-44f5-9cc3-cbaba2223946-maas-api-service-ca" pod "a7aa09dd-6974-44f5-9cc3-cbaba2223946" (UID: "a7aa09dd-6974-44f5-9cc3-cbaba2223946"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:19.880117 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.880059    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7aa09dd-6974-44f5-9cc3-cbaba2223946-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7aa09dd-6974-44f5-9cc3-cbaba2223946-maas-api-service-ca\") pod \"a7aa09dd-6974-44f5-9cc3-cbaba2223946\" (UID: \"a7aa09dd-6974-44f5-9cc3-cbaba2223946\") "
Oct 30 04:45:19.880376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.880203    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7aa09dd-6974-44f5-9cc3-cbaba2223946-kube-api-access-4x24z\" (UniqueName: \"kubernetes.io/projected/a7aa09dd-6974-44f5-9cc3-cbaba2223946-kube-api-access-4x24z\") pod \"a7aa09dd-6974-44f5-9cc3-cbaba2223946\" (UID: \"a7aa09dd-6974-44f5-9cc3-cbaba2223946\") "
Oct 30 04:45:19.882046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.881933    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7aa09dd-6974-44f5-9cc3-cbaba2223946-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:19.884724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.884682    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7aa09dd-6974-44f5-9cc3-cbaba2223946-kube-api-access-4x24z" pod "a7aa09dd-6974-44f5-9cc3-cbaba2223946" (UID: "a7aa09dd-6974-44f5-9cc3-cbaba2223946"). InnerVolumeSpecName "kube-api-access-4x24z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:19.962456 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.962428    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"
Oct 30 04:45:19.967480 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.967455    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"
Oct 30 04:45:19.973711 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.973685    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"
Oct 30 04:45:19.983349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.983321    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3abc0c26-3d30-4303-826f-7af0b072fc56-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3abc0c26-3d30-4303-826f-7af0b072fc56-maas-api-service-ca\") pod \"3abc0c26-3d30-4303-826f-7af0b072fc56\" (UID: \"3abc0c26-3d30-4303-826f-7af0b072fc56\") "
Oct 30 04:45:19.983476 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.983384    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d8b785-0448-4ed4-b17a-38d43794b893-kube-api-access-jsgrf\" (UniqueName: \"kubernetes.io/projected/a7d8b785-0448-4ed4-b17a-38d43794b893-kube-api-access-jsgrf\") pod \"a7d8b785-0448-4ed4-b17a-38d43794b893\" (UID: \"a7d8b785-0448-4ed4-b17a-38d43794b893\") "
Oct 30 04:45:19.983476 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.983445    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3abc0c26-3d30-4303-826f-7af0b072fc56-kube-api-access-jcf2c\" (UniqueName: \"kubernetes.io/projected/3abc0c26-3d30-4303-826f-7af0b072fc56-kube-api-access-jcf2c\") pod \"3abc0c26-3d30-4303-826f-7af0b072fc56\" (UID: \"3abc0c26-3d30-4303-826f-7af0b072fc56\") "
Oct 30 04:45:19.983568 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.983506    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25763423-975c-49f4-88a9-10b9e38015ea-kube-api-access-ldjg6\" (UniqueName: \"kubernetes.io/projected/25763423-975c-49f4-88a9-10b9e38015ea-kube-api-access-ldjg6\") pod \"25763423-975c-49f4-88a9-10b9e38015ea\" (UID: \"25763423-975c-49f4-88a9-10b9e38015ea\") "
Oct 30 04:45:19.983622 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.983584    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25763423-975c-49f4-88a9-10b9e38015ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25763423-975c-49f4-88a9-10b9e38015ea-maas-api-service-ca\") pod \"25763423-975c-49f4-88a9-10b9e38015ea\" (UID: \"25763423-975c-49f4-88a9-10b9e38015ea\") "
Oct 30 04:45:19.983702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.983677    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d8b785-0448-4ed4-b17a-38d43794b893-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8b785-0448-4ed4-b17a-38d43794b893-maas-api-service-ca\") pod \"a7d8b785-0448-4ed4-b17a-38d43794b893\" (UID: \"a7d8b785-0448-4ed4-b17a-38d43794b893\") "
Oct 30 04:45:19.984032 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.983676    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abc0c26-3d30-4303-826f-7af0b072fc56-maas-api-service-ca" pod "3abc0c26-3d30-4303-826f-7af0b072fc56" (UID: "3abc0c26-3d30-4303-826f-7af0b072fc56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:19.984174 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.984064    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25763423-975c-49f4-88a9-10b9e38015ea-maas-api-service-ca" pod "25763423-975c-49f4-88a9-10b9e38015ea" (UID: "25763423-975c-49f4-88a9-10b9e38015ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:19.984174 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.984151    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d8b785-0448-4ed4-b17a-38d43794b893-maas-api-service-ca" pod "a7d8b785-0448-4ed4-b17a-38d43794b893" (UID: "a7d8b785-0448-4ed4-b17a-38d43794b893"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:19.984472 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.984437    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8b785-0448-4ed4-b17a-38d43794b893-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:19.984472 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.984468    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x24z\" (UniqueName: \"kubernetes.io/projected/a7aa09dd-6974-44f5-9cc3-cbaba2223946-kube-api-access-4x24z\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:19.984592 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.984484    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3abc0c26-3d30-4303-826f-7af0b072fc56-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:19.984592 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.984498    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25763423-975c-49f4-88a9-10b9e38015ea-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:19.985887 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.985862    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abc0c26-3d30-4303-826f-7af0b072fc56-kube-api-access-jcf2c" pod "3abc0c26-3d30-4303-826f-7af0b072fc56" (UID: "3abc0c26-3d30-4303-826f-7af0b072fc56"). InnerVolumeSpecName "kube-api-access-jcf2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:19.986482 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.986451    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25763423-975c-49f4-88a9-10b9e38015ea-kube-api-access-ldjg6" pod "25763423-975c-49f4-88a9-10b9e38015ea" (UID: "25763423-975c-49f4-88a9-10b9e38015ea"). InnerVolumeSpecName "kube-api-access-ldjg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:19.986581 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:19.986550    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d8b785-0448-4ed4-b17a-38d43794b893-kube-api-access-jsgrf" pod "a7d8b785-0448-4ed4-b17a-38d43794b893" (UID: "a7d8b785-0448-4ed4-b17a-38d43794b893"). InnerVolumeSpecName "kube-api-access-jsgrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:20.085641 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.085594    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsgrf\" (UniqueName: \"kubernetes.io/projected/a7d8b785-0448-4ed4-b17a-38d43794b893-kube-api-access-jsgrf\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:20.085641 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.085630    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcf2c\" (UniqueName: \"kubernetes.io/projected/3abc0c26-3d30-4303-826f-7af0b072fc56-kube-api-access-jcf2c\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:20.085641 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.085645    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldjg6\" (UniqueName: \"kubernetes.io/projected/25763423-975c-49f4-88a9-10b9e38015ea-kube-api-access-ldjg6\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:20.610580 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.610503    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"
Oct 30 04:45:20.610580 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.610499    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824" event={"ID":"a7aa09dd-6974-44f5-9cc3-cbaba2223946","Type":"ContainerDied","Data":"8c8d0baefd7c6c0da788d1288986fe2d746c2ae2f0116f1d61a4ff3beed370fe"}
Oct 30 04:45:20.611098 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.610626    5072 scope.go:122] "RemoveContainer" containerID="aa81363115971dacf90a388913333ed01eac773299b6592c6654695a61a7795f"
Oct 30 04:45:20.615329 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.615299    5072 generic.go:356] "Generic (PLEG): container finished" podID="900d8674-41c1-434e-86cf-562bf6ef0822" containerID="4684b9cf730cf7fc9e3c4120b459ee6c3ec5a750b4688480dd3839b8e45ec7fa" exitCode=60
Oct 30 04:45:20.615463 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.615358    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx" event={"ID":"900d8674-41c1-434e-86cf-562bf6ef0822","Type":"ContainerDied","Data":"4684b9cf730cf7fc9e3c4120b459ee6c3ec5a750b4688480dd3839b8e45ec7fa"}
Oct 30 04:45:20.615760 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.615742    5072 scope.go:122] "RemoveContainer" containerID="4684b9cf730cf7fc9e3c4120b459ee6c3ec5a750b4688480dd3839b8e45ec7fa"
Oct 30 04:45:20.616071 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:20.616050    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx_odh-ai-gateway-infra(900d8674-41c1-434e-86cf-562bf6ef0822)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx" podUID="900d8674-41c1-434e-86cf-562bf6ef0822"
Oct 30 04:45:20.617050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.617024    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"
Oct 30 04:45:20.617186 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.617059    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86" event={"ID":"25763423-975c-49f4-88a9-10b9e38015ea","Type":"ContainerDied","Data":"6eccbd6a618b063f02eb80c151687ee1cf866297d0bd64251110b8ffdda0792d"}
Oct 30 04:45:20.619548 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.619521    5072 generic.go:356] "Generic (PLEG): container finished" podID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" containerID="cf40f36868bdd1d45599dc8dedf69a760b05200f88976be8197dd07b123f5f20" exitCode=60
Oct 30 04:45:20.619653 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.619586    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5" event={"ID":"5a984c77-fa0d-4970-a1a3-55351ab6e7ee","Type":"ContainerDied","Data":"cf40f36868bdd1d45599dc8dedf69a760b05200f88976be8197dd07b123f5f20"}
Oct 30 04:45:20.620112 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.620065    5072 scope.go:122] "RemoveContainer" containerID="cf40f36868bdd1d45599dc8dedf69a760b05200f88976be8197dd07b123f5f20"
Oct 30 04:45:20.620397 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:20.620372    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888925-m7sg5_odh-ai-gateway-infra(5a984c77-fa0d-4970-a1a3-55351ab6e7ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee"
Oct 30 04:45:20.621515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.621498    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"
Oct 30 04:45:20.621936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.621505    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2" event={"ID":"a7d8b785-0448-4ed4-b17a-38d43794b893","Type":"ContainerDied","Data":"d49bc9abccf557957bc1a39ca491c9b1d499b3f6a8419ca0f5af4d2df948a4cf"}
Oct 30 04:45:20.623136 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.623114    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx" event={"ID":"3abc0c26-3d30-4303-826f-7af0b072fc56","Type":"ContainerDied","Data":"cddc501b06e06353b5eb307de328e049528d909f8b954cab51dcfd503a0b4c89"}
Oct 30 04:45:20.623235 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.623127    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"
Oct 30 04:45:20.624420 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.624404    5072 scope.go:122] "RemoveContainer" containerID="b3a512fa5607e8d6526aecf62dbcfe5481ed0c9d7e5c050b42e7d3c912a4f019"
Oct 30 04:45:20.639640 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.639619    5072 scope.go:122] "RemoveContainer" containerID="aa393d87dab07093532c654ae84c94b41bdfad90b03442f901f115003076fe95"
Oct 30 04:45:20.650710 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.650687    5072 scope.go:122] "RemoveContainer" containerID="1a98182e71eade579f14e19fbfc0cc134b767e2dc86d00f97e6d530e80f658cd"
Oct 30 04:45:20.669914 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.669889    5072 scope.go:122] "RemoveContainer" containerID="9d193908d031f2264ad7239a8f6d2ac2371ab55a121aba2f895dc1f1c8a0892f"
Oct 30 04:45:20.692446 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.692404    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"]
Oct 30 04:45:20.692812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.692765    5072 scope.go:122] "RemoveContainer" containerID="32b19f1065e98d9bcd8324041549a8ae24af13f919704a5913dfb86aa1378238"
Oct 30 04:45:20.737096 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.733374    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"]
Oct 30 04:45:20.742224 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.740656    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888925-q8824"]
Oct 30 04:45:20.746275 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.744619    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"]
Oct 30 04:45:20.789217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.781151    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"]
Oct 30 04:45:20.789217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.788726    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888925-nqw86"]
Oct 30 04:45:20.814304 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.814274    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"
Oct 30 04:45:20.882564 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.882494    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"
Oct 30 04:45:20.896351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.896323    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eac636d9-a1c1-4220-974f-46a2666f0005-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac636d9-a1c1-4220-974f-46a2666f0005-maas-api-service-ca\") pod \"eac636d9-a1c1-4220-974f-46a2666f0005\" (UID: \"eac636d9-a1c1-4220-974f-46a2666f0005\") "
Oct 30 04:45:20.896532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.896363    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-kube-api-access-6cxkx\" (UniqueName: \"kubernetes.io/projected/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-kube-api-access-6cxkx\") pod \"2da7ba2d-6b50-4f01-8dc2-38552e35fd80\" (UID: \"2da7ba2d-6b50-4f01-8dc2-38552e35fd80\") "
Oct 30 04:45:20.896532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.896422    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-maas-api-service-ca\") pod \"2da7ba2d-6b50-4f01-8dc2-38552e35fd80\" (UID: \"2da7ba2d-6b50-4f01-8dc2-38552e35fd80\") "
Oct 30 04:45:20.896532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.896475    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eac636d9-a1c1-4220-974f-46a2666f0005-kube-api-access-hb2xh\" (UniqueName: \"kubernetes.io/projected/eac636d9-a1c1-4220-974f-46a2666f0005-kube-api-access-hb2xh\") pod \"eac636d9-a1c1-4220-974f-46a2666f0005\" (UID: \"eac636d9-a1c1-4220-974f-46a2666f0005\") "
Oct 30 04:45:20.896804 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.896752    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac636d9-a1c1-4220-974f-46a2666f0005-maas-api-service-ca" pod "eac636d9-a1c1-4220-974f-46a2666f0005" (UID: "eac636d9-a1c1-4220-974f-46a2666f0005"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:20.896878 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.896806    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-maas-api-service-ca" pod "2da7ba2d-6b50-4f01-8dc2-38552e35fd80" (UID: "2da7ba2d-6b50-4f01-8dc2-38552e35fd80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:20.896943 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.896927    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac636d9-a1c1-4220-974f-46a2666f0005-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:20.896985 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.896949    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:20.898879 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.898855    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-kube-api-access-6cxkx" pod "2da7ba2d-6b50-4f01-8dc2-38552e35fd80" (UID: "2da7ba2d-6b50-4f01-8dc2-38552e35fd80"). InnerVolumeSpecName "kube-api-access-6cxkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:20.898982 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.898928    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac636d9-a1c1-4220-974f-46a2666f0005-kube-api-access-hb2xh" pod "eac636d9-a1c1-4220-974f-46a2666f0005" (UID: "eac636d9-a1c1-4220-974f-46a2666f0005"). InnerVolumeSpecName "kube-api-access-hb2xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:20.918915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.918877    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"]
Oct 30 04:45:20.924385 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.924359    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888925-tjct2"]
Oct 30 04:45:20.998019 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.997975    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cxkx\" (UniqueName: \"kubernetes.io/projected/2da7ba2d-6b50-4f01-8dc2-38552e35fd80-kube-api-access-6cxkx\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:20.998019 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:20.998020    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb2xh\" (UniqueName: \"kubernetes.io/projected/eac636d9-a1c1-4220-974f-46a2666f0005-kube-api-access-hb2xh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:21.042787 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.042556    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"]
Oct 30 04:45:21.046706 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.046669    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888925-4jqhx"]
Oct 30 04:45:21.634410 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.634378    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"
Oct 30 04:45:21.634836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.634377    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv" event={"ID":"eac636d9-a1c1-4220-974f-46a2666f0005","Type":"ContainerDied","Data":"3531c6d767667636b3007ca262d6fdb56b543ec8b539f5dbe80cbc9ed6eee015"}
Oct 30 04:45:21.634836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.634558    5072 scope.go:122] "RemoveContainer" containerID="496a93637872c60c975870241c30473e5eb0ddd1225c0180a35563944d884921"
Oct 30 04:45:21.645309 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.645288    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"
Oct 30 04:45:21.645309 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.645287    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt" event={"ID":"2da7ba2d-6b50-4f01-8dc2-38552e35fd80","Type":"ContainerDied","Data":"aa0c014402907c49805f0e03003499b5597732b0fb2ded007f3806d857c873a2"}
Oct 30 04:45:21.647411 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.647390    5072 scope.go:122] "RemoveContainer" containerID="1a9c49c0b17cba8c826be5221cc569cc610b4619e8e1d15b6e76ebf9c3e852a0"
Oct 30 04:45:21.750799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.750766    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"]
Oct 30 04:45:21.755955 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.755919    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888925-zt8wv"]
Oct 30 04:45:21.787532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.787494    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"]
Oct 30 04:45:21.794599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.794569    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888925-fj2nt"]
Oct 30 04:45:21.797677 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.797644    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"
Oct 30 04:45:21.807516 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.807488    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900d8674-41c1-434e-86cf-562bf6ef0822-kube-api-access-mbfr2\" (UniqueName: \"kubernetes.io/projected/900d8674-41c1-434e-86cf-562bf6ef0822-kube-api-access-mbfr2\") pod \"900d8674-41c1-434e-86cf-562bf6ef0822\" (UID: \"900d8674-41c1-434e-86cf-562bf6ef0822\") "
Oct 30 04:45:21.807687 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.807584    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900d8674-41c1-434e-86cf-562bf6ef0822-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/900d8674-41c1-434e-86cf-562bf6ef0822-maas-api-service-ca\") pod \"900d8674-41c1-434e-86cf-562bf6ef0822\" (UID: \"900d8674-41c1-434e-86cf-562bf6ef0822\") "
Oct 30 04:45:21.807975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.807948    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900d8674-41c1-434e-86cf-562bf6ef0822-maas-api-service-ca" pod "900d8674-41c1-434e-86cf-562bf6ef0822" (UID: "900d8674-41c1-434e-86cf-562bf6ef0822"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:21.809910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.809885    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900d8674-41c1-434e-86cf-562bf6ef0822-kube-api-access-mbfr2" pod "900d8674-41c1-434e-86cf-562bf6ef0822" (UID: "900d8674-41c1-434e-86cf-562bf6ef0822"). InnerVolumeSpecName "kube-api-access-mbfr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:21.850157 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.850120    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"]
Oct 30 04:45:21.908770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.908735    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbfr2\" (UniqueName: \"kubernetes.io/projected/900d8674-41c1-434e-86cf-562bf6ef0822-kube-api-access-mbfr2\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:21.908770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:21.908766    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/900d8674-41c1-434e-86cf-562bf6ef0822-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:22.028645 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.028616    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"
Oct 30 04:45:22.110789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.110753    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-maas-api-service-ca\") pod \"5a984c77-fa0d-4970-a1a3-55351ab6e7ee\" (UID: \"5a984c77-fa0d-4970-a1a3-55351ab6e7ee\") "
Oct 30 04:45:22.111006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.110903    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-kube-api-access-c9cvk\" (UniqueName: \"kubernetes.io/projected/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-kube-api-access-c9cvk\") pod \"5a984c77-fa0d-4970-a1a3-55351ab6e7ee\" (UID: \"5a984c77-fa0d-4970-a1a3-55351ab6e7ee\") "
Oct 30 04:45:22.111129 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.111105    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-maas-api-service-ca" pod "5a984c77-fa0d-4970-a1a3-55351ab6e7ee" (UID: "5a984c77-fa0d-4970-a1a3-55351ab6e7ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:22.111478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.111457    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:22.113356 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.113330    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-kube-api-access-c9cvk" pod "5a984c77-fa0d-4970-a1a3-55351ab6e7ee" (UID: "5a984c77-fa0d-4970-a1a3-55351ab6e7ee"). InnerVolumeSpecName "kube-api-access-c9cvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:22.212292 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.212261    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9cvk\" (UniqueName: \"kubernetes.io/projected/5a984c77-fa0d-4970-a1a3-55351ab6e7ee-kube-api-access-c9cvk\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:45:22.409817 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.409736    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25763423-975c-49f4-88a9-10b9e38015ea" path="/var/lib/kubelet/pods/25763423-975c-49f4-88a9-10b9e38015ea/volumes"
Oct 30 04:45:22.410232 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.410218    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" path="/var/lib/kubelet/pods/2da7ba2d-6b50-4f01-8dc2-38552e35fd80/volumes"
Oct 30 04:45:22.410579 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.410568    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56" path="/var/lib/kubelet/pods/3abc0c26-3d30-4303-826f-7af0b072fc56/volumes"
Oct 30 04:45:22.411255 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.411240    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" path="/var/lib/kubelet/pods/a7aa09dd-6974-44f5-9cc3-cbaba2223946/volumes"
Oct 30 04:45:22.411619 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.411604    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893" path="/var/lib/kubelet/pods/a7d8b785-0448-4ed4-b17a-38d43794b893/volumes"
Oct 30 04:45:22.411974 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.411960    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eac636d9-a1c1-4220-974f-46a2666f0005" path="/var/lib/kubelet/pods/eac636d9-a1c1-4220-974f-46a2666f0005/volumes"
Oct 30 04:45:22.655720 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.655686    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"
Oct 30 04:45:22.655720 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.655713    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx" event={"ID":"900d8674-41c1-434e-86cf-562bf6ef0822","Type":"ContainerDied","Data":"b71593fae05cadc9d3d128434387a0ca84bb1edd4416d57e7065406aea9fac36"}
Oct 30 04:45:22.656296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.655779    5072 scope.go:122] "RemoveContainer" containerID="4684b9cf730cf7fc9e3c4120b459ee6c3ec5a750b4688480dd3839b8e45ec7fa"
Oct 30 04:45:22.657856 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.657463    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"
Oct 30 04:45:22.657856 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.657461    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5" event={"ID":"5a984c77-fa0d-4970-a1a3-55351ab6e7ee","Type":"ContainerDied","Data":"238663de760cc44383be1e2a41b182115c9b0547aafba384ea4e0320e5bba8f1"}
Oct 30 04:45:22.667138 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.667115    5072 scope.go:122] "RemoveContainer" containerID="cf40f36868bdd1d45599dc8dedf69a760b05200f88976be8197dd07b123f5f20"
Oct 30 04:45:22.710979 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.710941    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"]
Oct 30 04:45:22.715145 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.715120    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888925-m7sg5"]
Oct 30 04:45:22.753812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.753771    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"]
Oct 30 04:45:22.761327 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:22.761295    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888925-gnskx"]
Oct 30 04:45:24.409881 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:24.409850    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" path="/var/lib/kubelet/pods/5a984c77-fa0d-4970-a1a3-55351ab6e7ee/volumes"
Oct 30 04:45:24.410349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:45:24.410334    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900d8674-41c1-434e-86cf-562bf6ef0822" path="/var/lib/kubelet/pods/900d8674-41c1-434e-86cf-562bf6ef0822/volumes"
Oct 30 04:45:41.752178 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:45:41.752069    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:46:03.875244 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.875197    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"]
Oct 30 04:46:03.876795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876756    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893" containerName="cleanup"
Oct 30 04:46:03.876795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876788    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876820    5072 cpu_manager.go:414] "containerMap: removing container" podUID="25763423-975c-49f4-88a9-10b9e38015ea" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876829    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25763423-975c-49f4-88a9-10b9e38015ea" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876841    5072 cpu_manager.go:414] "containerMap: removing container" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876848    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876862    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876870    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876886    5072 cpu_manager.go:414] "containerMap: removing container" podUID="eac636d9-a1c1-4220-974f-46a2666f0005" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876893    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac636d9-a1c1-4220-974f-46a2666f0005" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876902    5072 cpu_manager.go:414] "containerMap: removing container" podUID="eac636d9-a1c1-4220-974f-46a2666f0005" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876909    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac636d9-a1c1-4220-974f-46a2666f0005" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876921    5072 cpu_manager.go:414] "containerMap: removing container" podUID="52403e2b-a240-452e-98f1-c577b1f770db" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876928    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52403e2b-a240-452e-98f1-c577b1f770db" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876943    5072 cpu_manager.go:414] "containerMap: removing container" podUID="900d8674-41c1-434e-86cf-562bf6ef0822" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876952    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900d8674-41c1-434e-86cf-562bf6ef0822" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876964    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876970    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876987    5072 cpu_manager.go:414] "containerMap: removing container" podUID="8b1ad519-9858-4903-847c-e8dc14645411" containerName="main"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.876993    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1ad519-9858-4903-847c-e8dc14645411" containerName="main"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877003    5072 cpu_manager.go:414] "containerMap: removing container" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877007    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56" containerName="cleanup"
Oct 30 04:46:03.877010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877024    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877028    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877034    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877039    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877050    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877056    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877069    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877095    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877104    5072 cpu_manager.go:414] "containerMap: removing container" podUID="900d8674-41c1-434e-86cf-562bf6ef0822" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877110    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900d8674-41c1-434e-86cf-562bf6ef0822" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877122    5072 cpu_manager.go:414] "containerMap: removing container" podUID="900d8674-41c1-434e-86cf-562bf6ef0822" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877129    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900d8674-41c1-434e-86cf-562bf6ef0822" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877135    5072 cpu_manager.go:414] "containerMap: removing container" podUID="52403e2b-a240-452e-98f1-c577b1f770db" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877138    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52403e2b-a240-452e-98f1-c577b1f770db" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877146    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877149    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877160    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877164    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877170    5072 cpu_manager.go:414] "containerMap: removing container" podUID="25763423-975c-49f4-88a9-10b9e38015ea" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877175    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25763423-975c-49f4-88a9-10b9e38015ea" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877183    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877189    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877325    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877338    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877348    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="900d8674-41c1-434e-86cf-562bf6ef0822" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877358    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="900d8674-41c1-434e-86cf-562bf6ef0822" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877368    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="52403e2b-a240-452e-98f1-c577b1f770db" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877378    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877387    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac636d9-a1c1-4220-974f-46a2666f0005" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877396    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877405    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877414    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7aa09dd-6974-44f5-9cc3-cbaba2223946" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877426    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="25763423-975c-49f4-88a9-10b9e38015ea" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877436    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877447    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877459    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877469    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877480    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac636d9-a1c1-4220-974f-46a2666f0005" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877488    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac636d9-a1c1-4220-974f-46a2666f0005" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877499    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="52403e2b-a240-452e-98f1-c577b1f770db" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877509    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877517    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1ad519-9858-4903-847c-e8dc14645411" containerName="main"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877524    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877533    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="25763423-975c-49f4-88a9-10b9e38015ea" containerName="cleanup"
Oct 30 04:46:03.878140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.877538    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893" containerName="cleanup"
Oct 30 04:46:03.883800 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.883779    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:03.914010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.913003    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-503804-openshift-default-dockercfg-gsmss\""
Oct 30 04:46:03.918315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.918289    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"]
Oct 30 04:46:03.982207 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.982176    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnk8w\" (UniqueName: \"kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-kube-api-access-mnk8w\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:03.982366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.982319    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-credential-socket\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:03.982418 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.982368    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-podinfo\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:03.982418 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.982385    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-token\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:03.982506 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.982487    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istiod-ca-cert\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:03.982542 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.982528    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-socket\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:03.982578 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.982550    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-certs\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:03.982614 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.982598    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-envoy\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:03.982745 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:03.982721    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-data\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.084272 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.084234    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-podinfo\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.084460 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.084285    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-token\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.084460 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.084350    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istiod-ca-cert\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.084589 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.084514    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-socket\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.084589 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.084570    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-certs\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.084701 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.084630    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-envoy\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.084701 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.084683    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-data\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.084805 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.084756    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnk8w\" (UniqueName: \"kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-kube-api-access-mnk8w\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.084925 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.084902    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-credential-socket\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.085250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.085199    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istiod-ca-cert\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.085250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.085108    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-certs\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.085250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.085231    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-data\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.085487 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.084915    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-socket\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.085882 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.085857    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-credential-socket\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.087004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.086981    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-envoy\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.087109 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.087047    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-podinfo\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.114823 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.114792    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnk8w\" (UniqueName: \"kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-kube-api-access-mnk8w\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.115362 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.115338    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-token\") pod \"e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") " pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.201717 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.201684    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:04.526337 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.526307    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"]
Oct 30 04:46:04.526523 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:46:04.526501    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf86bd0_b7e9_43ae_a7e3_ac5841b92af9.slice/crio-42446a50d1a1fbcef9cbc8fa78826a72883e810227edf1f9357c86e21858f95f WatchSource:0}: Error finding container 42446a50d1a1fbcef9cbc8fa78826a72883e810227edf1f9357c86e21858f95f: Status 404 returned error can't find the container with id 42446a50d1a1fbcef9cbc8fa78826a72883e810227edf1f9357c86e21858f95f
Oct 30 04:46:04.529906 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.529864    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:46:04.530024 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.529945    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:46:04.530024 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.529982    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:46:04.954335 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.954292    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" event={"ID":"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9","Type":"ContainerStarted","Data":"84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0"}
Oct 30 04:46:04.954780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:04.954361    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" event={"ID":"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9","Type":"ContainerStarted","Data":"42446a50d1a1fbcef9cbc8fa78826a72883e810227edf1f9357c86e21858f95f"}
Oct 30 04:46:05.006373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:05.006326    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" podStartSLOduration=2.006311324 podStartE2EDuration="2.006311324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:05.003010845 +0000 UTC m=+2944.744433942" watchObservedRunningTime="2026-10-30 04:46:05.006311324 +0000 UTC m=+2944.747734531"
Oct 30 04:46:05.202859 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:05.202821    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:05.209303 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:05.209226    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:05.963251 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:05.963223    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:05.964378 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:05.964350    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:46:06.810887 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.810854    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"]
Oct 30 04:46:06.811501 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811487    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" containerName="cleanup"
Oct 30 04:46:06.811553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811503    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a984c77-fa0d-4970-a1a3-55351ab6e7ee" containerName="cleanup"
Oct 30 04:46:06.811553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811516    5072 cpu_manager.go:414] "containerMap: removing container" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56" containerName="cleanup"
Oct 30 04:46:06.811553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811521    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abc0c26-3d30-4303-826f-7af0b072fc56" containerName="cleanup"
Oct 30 04:46:06.811553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811544    5072 cpu_manager.go:414] "containerMap: removing container" podUID="52403e2b-a240-452e-98f1-c577b1f770db" containerName="cleanup"
Oct 30 04:46:06.811553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811549    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52403e2b-a240-452e-98f1-c577b1f770db" containerName="cleanup"
Oct 30 04:46:06.811704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811562    5072 cpu_manager.go:414] "containerMap: removing container" podUID="25763423-975c-49f4-88a9-10b9e38015ea" containerName="cleanup"
Oct 30 04:46:06.811704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811567    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25763423-975c-49f4-88a9-10b9e38015ea" containerName="cleanup"
Oct 30 04:46:06.811704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811589    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" containerName="cleanup"
Oct 30 04:46:06.811704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811594    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" containerName="cleanup"
Oct 30 04:46:06.811704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811603    5072 cpu_manager.go:414] "containerMap: removing container" podUID="eac636d9-a1c1-4220-974f-46a2666f0005" containerName="cleanup"
Oct 30 04:46:06.811704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811607    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac636d9-a1c1-4220-974f-46a2666f0005" containerName="cleanup"
Oct 30 04:46:06.811704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811690    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da7ba2d-6b50-4f01-8dc2-38552e35fd80" containerName="cleanup"
Oct 30 04:46:06.811704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811705    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="900d8674-41c1-434e-86cf-562bf6ef0822" containerName="cleanup"
Oct 30 04:46:06.811928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811711    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="52403e2b-a240-452e-98f1-c577b1f770db" containerName="cleanup"
Oct 30 04:46:06.811928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811719    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="25763423-975c-49f4-88a9-10b9e38015ea" containerName="cleanup"
Oct 30 04:46:06.811928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.811726    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d8b785-0448-4ed4-b17a-38d43794b893" containerName="cleanup"
Oct 30 04:46:06.815296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.815278    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:06.833620 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.833596    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"]
Oct 30 04:46:06.843149 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.843122    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"routing-overlay\""
Oct 30 04:46:06.843428 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.843407    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"agc-cred-a5e8b03626c3\""
Oct 30 04:46:06.843587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.843463    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"payload-processing-external-model-plugins-e2e-extmod-b-483bea\""
Oct 30 04:46:06.843705 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.843607    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"kube-root-ca.crt\""
Oct 30 04:46:06.845531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.845514    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"openshift-service-ca.crt\""
Oct 30 04:46:06.845648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.845597    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"payload-processing-external-model-e2e--675c86f9-dockercfg-czldl\""
Oct 30 04:46:06.920992 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.920953    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvjd7\" (UniqueName: \"kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-kube-api-access-hvjd7\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:06.921173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.921001    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:06.921220 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.921174    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:06.921274 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:06.921230    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:07.021969 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:07.021927    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:07.022501 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:07.022142    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvjd7\" (UniqueName: \"kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-kube-api-access-hvjd7\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:07.022501 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:07.022197    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:07.022501 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:07.022284    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:07.023291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:07.023257    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:07.023492 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:07.023386    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:07.025991 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:07.025961    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:07.046821 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:07.046782    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvjd7\" (UniqueName: \"kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-kube-api-access-hvjd7\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:07.130782 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:07.130712    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:07.438779 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:46:07.438749    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3a1d2a_bd77_4141_a4c2_ad672da165ee.slice/crio-7ff1e0ba240ef0a5b7991f72d59773d96e3ce58432727ab86ea4f6c3c476a81a WatchSource:0}: Error finding container 7ff1e0ba240ef0a5b7991f72d59773d96e3ce58432727ab86ea4f6c3c476a81a: Status 404 returned error can't find the container with id 7ff1e0ba240ef0a5b7991f72d59773d96e3ce58432727ab86ea4f6c3c476a81a
Oct 30 04:46:07.440577 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:07.440472    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"]
Oct 30 04:46:07.983131 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:07.983098    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7" event={"ID":"bd3a1d2a-bd77-4141-a4c2-ad672da165ee","Type":"ContainerStarted","Data":"3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132"}
Oct 30 04:46:07.983283 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:07.983148    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7" event={"ID":"bd3a1d2a-bd77-4141-a4c2-ad672da165ee","Type":"ContainerStarted","Data":"7ff1e0ba240ef0a5b7991f72d59773d96e3ce58432727ab86ea4f6c3c476a81a"}
Oct 30 04:46:08.028022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:08.027968    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7" podStartSLOduration=2.02795517 podStartE2EDuration="2.02795517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:08.024669432 +0000 UTC m=+2947.766092529" watchObservedRunningTime="2026-10-30 04:46:08.02795517 +0000 UTC m=+2947.769378265"
Oct 30 04:46:08.991252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:08.991207    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:09.486427 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.486394    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"]
Oct 30 04:46:09.491105 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.491071    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"
Oct 30 04:46:09.516117 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.516090    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"]
Oct 30 04:46:09.526095 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.523692    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-503804\""
Oct 30 04:46:09.526095 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.523777    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-503804-dockercfg-sqpw4\""
Oct 30 04:46:09.580951 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.580909    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2csfr\" (UniqueName: \"kubernetes.io/projected/620fa980-d930-44fe-b306-5db27ed9e675-kube-api-access-2csfr\") pod \"payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx\" (UID: \"620fa980-d930-44fe-b306-5db27ed9e675\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"
Oct 30 04:46:09.582453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.582418    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620fa980-d930-44fe-b306-5db27ed9e675-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx\" (UID: \"620fa980-d930-44fe-b306-5db27ed9e675\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"
Oct 30 04:46:09.585297 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.585268    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"]
Oct 30 04:46:09.590699 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.590671    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"
Oct 30 04:46:09.631697 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.631660    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"]
Oct 30 04:46:09.684232 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.684192    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cq5c\" (UniqueName: \"kubernetes.io/projected/ee7d87cd-c93b-49d6-8319-af928d231319-kube-api-access-8cq5c\") pod \"payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2\" (UID: \"ee7d87cd-c93b-49d6-8319-af928d231319\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"
Oct 30 04:46:09.684420 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.684254    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee7d87cd-c93b-49d6-8319-af928d231319-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2\" (UID: \"ee7d87cd-c93b-49d6-8319-af928d231319\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"
Oct 30 04:46:09.684420 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.684359    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2csfr\" (UniqueName: \"kubernetes.io/projected/620fa980-d930-44fe-b306-5db27ed9e675-kube-api-access-2csfr\") pod \"payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx\" (UID: \"620fa980-d930-44fe-b306-5db27ed9e675\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"
Oct 30 04:46:09.684420 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.684395    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620fa980-d930-44fe-b306-5db27ed9e675-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx\" (UID: \"620fa980-d930-44fe-b306-5db27ed9e675\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"
Oct 30 04:46:09.685416 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.685389    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620fa980-d930-44fe-b306-5db27ed9e675-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx\" (UID: \"620fa980-d930-44fe-b306-5db27ed9e675\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"
Oct 30 04:46:09.709037 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.709004    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2csfr\" (UniqueName: \"kubernetes.io/projected/620fa980-d930-44fe-b306-5db27ed9e675-kube-api-access-2csfr\") pod \"payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx\" (UID: \"620fa980-d930-44fe-b306-5db27ed9e675\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"
Oct 30 04:46:09.785520 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.785447    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cq5c\" (UniqueName: \"kubernetes.io/projected/ee7d87cd-c93b-49d6-8319-af928d231319-kube-api-access-8cq5c\") pod \"payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2\" (UID: \"ee7d87cd-c93b-49d6-8319-af928d231319\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"
Oct 30 04:46:09.785520 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.785496    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee7d87cd-c93b-49d6-8319-af928d231319-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2\" (UID: \"ee7d87cd-c93b-49d6-8319-af928d231319\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"
Oct 30 04:46:09.786320 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.786295    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee7d87cd-c93b-49d6-8319-af928d231319-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2\" (UID: \"ee7d87cd-c93b-49d6-8319-af928d231319\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"
Oct 30 04:46:09.806225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.806189    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"
Oct 30 04:46:09.809891 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.809874    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cq5c\" (UniqueName: \"kubernetes.io/projected/ee7d87cd-c93b-49d6-8319-af928d231319-kube-api-access-8cq5c\") pod \"payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2\" (UID: \"ee7d87cd-c93b-49d6-8319-af928d231319\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"
Oct 30 04:46:09.907664 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:09.907614    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"
Oct 30 04:46:10.111458 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:46:10.111428    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620fa980_d930_44fe_b306_5db27ed9e675.slice/crio-0f22de327a9dc0218c6f0e10cbf20e3d1f599ac0cdd0017474c8438e940d87ff WatchSource:0}: Error finding container 0f22de327a9dc0218c6f0e10cbf20e3d1f599ac0cdd0017474c8438e940d87ff: Status 404 returned error can't find the container with id 0f22de327a9dc0218c6f0e10cbf20e3d1f599ac0cdd0017474c8438e940d87ff
Oct 30 04:46:10.112542 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:10.112487    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"]
Oct 30 04:46:10.227331 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:46:10.227293    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7d87cd_c93b_49d6_8319_af928d231319.slice/crio-f8ec07035ef79658f5ad7a0f3d564cacdd1f7ec5fc0c4bc834b68bfeb9c76002 WatchSource:0}: Error finding container f8ec07035ef79658f5ad7a0f3d564cacdd1f7ec5fc0c4bc834b68bfeb9c76002: Status 404 returned error can't find the container with id f8ec07035ef79658f5ad7a0f3d564cacdd1f7ec5fc0c4bc834b68bfeb9c76002
Oct 30 04:46:10.228412 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:10.228386    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"]
Oct 30 04:46:11.009888 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:11.009847    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2" event={"ID":"ee7d87cd-c93b-49d6-8319-af928d231319","Type":"ContainerStarted","Data":"4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d"}
Oct 30 04:46:11.010332 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:11.009914    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2" event={"ID":"ee7d87cd-c93b-49d6-8319-af928d231319","Type":"ContainerStarted","Data":"f8ec07035ef79658f5ad7a0f3d564cacdd1f7ec5fc0c4bc834b68bfeb9c76002"}
Oct 30 04:46:11.010332 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:11.010054    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"
Oct 30 04:46:11.011670 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:11.011632    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx" event={"ID":"620fa980-d930-44fe-b306-5db27ed9e675","Type":"ContainerStarted","Data":"00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a"}
Oct 30 04:46:11.011797 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:11.011688    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx" event={"ID":"620fa980-d930-44fe-b306-5db27ed9e675","Type":"ContainerStarted","Data":"0f22de327a9dc0218c6f0e10cbf20e3d1f599ac0cdd0017474c8438e940d87ff"}
Oct 30 04:46:11.011797 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:11.011719    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"
Oct 30 04:46:11.056242 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:11.056193    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2" podStartSLOduration=2.056179436 podStartE2EDuration="2.056179436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:11.053135407 +0000 UTC m=+2950.794558505" watchObservedRunningTime="2026-10-30 04:46:11.056179436 +0000 UTC m=+2950.797602530"
Oct 30 04:46:11.096508 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:11.096461    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx" podStartSLOduration=2.096447699 podStartE2EDuration="2.096447699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:11.093933412 +0000 UTC m=+2950.835356510" watchObservedRunningTime="2026-10-30 04:46:11.096447699 +0000 UTC m=+2950.837870787"
Oct 30 04:46:11.752282 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:46:11.752246    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:46:13.044053 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:13.044018    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11264-6669e7"]
Oct 30 04:46:13.048580 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:13.048555    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11264-6669e7"
Oct 30 04:46:13.063282 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:13.063258    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11264-6669e7"]
Oct 30 04:46:13.071670 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:13.071649    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gsgvm\""
Oct 30 04:46:13.229027 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:13.228997    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5hg7\" (UniqueName: \"kubernetes.io/projected/563307c2-9b8f-4666-9f20-09459c7b7b70-kube-api-access-n5hg7\") pod \"test-curl-11264-6669e7\" (UID: \"563307c2-9b8f-4666-9f20-09459c7b7b70\") " pod="opendatahub/test-curl-11264-6669e7"
Oct 30 04:46:13.329921 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:13.329835    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5hg7\" (UniqueName: \"kubernetes.io/projected/563307c2-9b8f-4666-9f20-09459c7b7b70-kube-api-access-n5hg7\") pod \"test-curl-11264-6669e7\" (UID: \"563307c2-9b8f-4666-9f20-09459c7b7b70\") " pod="opendatahub/test-curl-11264-6669e7"
Oct 30 04:46:13.354260 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:13.354234    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5hg7\" (UniqueName: \"kubernetes.io/projected/563307c2-9b8f-4666-9f20-09459c7b7b70-kube-api-access-n5hg7\") pod \"test-curl-11264-6669e7\" (UID: \"563307c2-9b8f-4666-9f20-09459c7b7b70\") " pod="opendatahub/test-curl-11264-6669e7"
Oct 30 04:46:13.363352 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:13.363319    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11264-6669e7"
Oct 30 04:46:13.667292 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:46:13.667250    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod563307c2_9b8f_4666_9f20_09459c7b7b70.slice/crio-66be050b9ccfdf0c5fbe284a1f9e7337ab86b964d6b2dce687d4c64d10e9ebc4 WatchSource:0}: Error finding container 66be050b9ccfdf0c5fbe284a1f9e7337ab86b964d6b2dce687d4c64d10e9ebc4: Status 404 returned error can't find the container with id 66be050b9ccfdf0c5fbe284a1f9e7337ab86b964d6b2dce687d4c64d10e9ebc4
Oct 30 04:46:13.667517 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:13.667497    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11264-6669e7"]
Oct 30 04:46:14.033665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.033624    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11264-6669e7" event={"ID":"563307c2-9b8f-4666-9f20-09459c7b7b70","Type":"ContainerStarted","Data":"e255aafb252eaed8241072290539c1cf12f68d2460740bcacf4e17687b7afdc4"}
Oct 30 04:46:14.033880 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.033675    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11264-6669e7" event={"ID":"563307c2-9b8f-4666-9f20-09459c7b7b70","Type":"ContainerStarted","Data":"66be050b9ccfdf0c5fbe284a1f9e7337ab86b964d6b2dce687d4c64d10e9ebc4"}
Oct 30 04:46:14.076598 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.076545    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11264-6669e7" podStartSLOduration=1.07653235 podStartE2EDuration="1.07653235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:14.075764875 +0000 UTC m=+2953.817187972" watchObservedRunningTime="2026-10-30 04:46:14.07653235 +0000 UTC m=+2953.817955446"
Oct 30 04:46:14.439343 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.439315    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"]
Oct 30 04:46:14.443961 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.443940    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"
Oct 30 04:46:14.461181 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.461143    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b195d0\""
Oct 30 04:46:14.463022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.462995    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"]
Oct 30 04:46:14.543791 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.543755    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x455\" (UniqueName: \"kubernetes.io/projected/f8ff9e15-586c-4b92-9e47-7d79149ebe93-kube-api-access-4x455\") pod \"payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4\" (UID: \"f8ff9e15-586c-4b92-9e47-7d79149ebe93\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"
Oct 30 04:46:14.543972 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.543797    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ff9e15-586c-4b92-9e47-7d79149ebe93-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4\" (UID: \"f8ff9e15-586c-4b92-9e47-7d79149ebe93\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"
Oct 30 04:46:14.646144 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.645373    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x455\" (UniqueName: \"kubernetes.io/projected/f8ff9e15-586c-4b92-9e47-7d79149ebe93-kube-api-access-4x455\") pod \"payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4\" (UID: \"f8ff9e15-586c-4b92-9e47-7d79149ebe93\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"
Oct 30 04:46:14.646144 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.645443    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ff9e15-586c-4b92-9e47-7d79149ebe93-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4\" (UID: \"f8ff9e15-586c-4b92-9e47-7d79149ebe93\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"
Oct 30 04:46:14.647093 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.647048    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ff9e15-586c-4b92-9e47-7d79149ebe93-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4\" (UID: \"f8ff9e15-586c-4b92-9e47-7d79149ebe93\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"
Oct 30 04:46:14.673880 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.673848    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x455\" (UniqueName: \"kubernetes.io/projected/f8ff9e15-586c-4b92-9e47-7d79149ebe93-kube-api-access-4x455\") pod \"payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4\" (UID: \"f8ff9e15-586c-4b92-9e47-7d79149ebe93\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"
Oct 30 04:46:14.759255 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.759182    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"
Oct 30 04:46:14.970163 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.970128    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11264-6669e7"]
Oct 30 04:46:14.987788 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:14.987758    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11264-6669e7"]
Oct 30 04:46:15.072920 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:46:15.072895    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ff9e15_586c_4b92_9e47_7d79149ebe93.slice/crio-4d655e3fc39cdf3c33fb68a90efe8ef2b5f3eb1b9700ca69d01e9b497508fb1d WatchSource:0}: Error finding container 4d655e3fc39cdf3c33fb68a90efe8ef2b5f3eb1b9700ca69d01e9b497508fb1d: Status 404 returned error can't find the container with id 4d655e3fc39cdf3c33fb68a90efe8ef2b5f3eb1b9700ca69d01e9b497508fb1d
Oct 30 04:46:15.073609 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.073584    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"]
Oct 30 04:46:15.192409 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.192375    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"]
Oct 30 04:46:15.193196 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.193061    5072 cpu_manager.go:414] "containerMap: removing container" podUID="563307c2-9b8f-4666-9f20-09459c7b7b70" containerName="test-curl-11264-6669e7"
Oct 30 04:46:15.193196 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.193097    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="563307c2-9b8f-4666-9f20-09459c7b7b70" containerName="test-curl-11264-6669e7"
Oct 30 04:46:15.193324 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.193273    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="563307c2-9b8f-4666-9f20-09459c7b7b70" containerName="test-curl-11264-6669e7"
Oct 30 04:46:15.196863 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.196848    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:15.216681 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.216653    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"]
Oct 30 04:46:15.253206 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.253173    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:46:15.253371 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.253207    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:46:15.356562 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.356475    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ebcf114-hp4rg\" (UID: \"394b0d01-4520-4eb6-8490-719818e3ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:15.356562 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.356518    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqf96\" (UniqueName: \"kubernetes.io/projected/394b0d01-4520-4eb6-8490-719818e3ad43-kube-api-access-qqf96\") pod \"maas-api-revoke-sub-7ebcf114-hp4rg\" (UID: \"394b0d01-4520-4eb6-8490-719818e3ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:15.356852 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.356682    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ebcf114-hp4rg\" (UID: \"394b0d01-4520-4eb6-8490-719818e3ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:15.457931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.457883    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ebcf114-hp4rg\" (UID: \"394b0d01-4520-4eb6-8490-719818e3ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:15.458136 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.457960    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqf96\" (UniqueName: \"kubernetes.io/projected/394b0d01-4520-4eb6-8490-719818e3ad43-kube-api-access-qqf96\") pod \"maas-api-revoke-sub-7ebcf114-hp4rg\" (UID: \"394b0d01-4520-4eb6-8490-719818e3ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:15.458284 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.458103    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ebcf114-hp4rg\" (UID: \"394b0d01-4520-4eb6-8490-719818e3ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:15.458624 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.458594    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ebcf114-hp4rg\" (UID: \"394b0d01-4520-4eb6-8490-719818e3ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:15.461421 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.461401    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ebcf114-hp4rg\" (UID: \"394b0d01-4520-4eb6-8490-719818e3ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:15.482133 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.482106    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqf96\" (UniqueName: \"kubernetes.io/projected/394b0d01-4520-4eb6-8490-719818e3ad43-kube-api-access-qqf96\") pod \"maas-api-revoke-sub-7ebcf114-hp4rg\" (UID: \"394b0d01-4520-4eb6-8490-719818e3ad43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:15.511305 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.511281    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:15.816373 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:46:15.816342    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394b0d01_4520_4eb6_8490_719818e3ad43.slice/crio-0241aeae90ccba41107941f724f47941e7a33f7948d75605bfbfb7ce7a8e746c WatchSource:0}: Error finding container 0241aeae90ccba41107941f724f47941e7a33f7948d75605bfbfb7ce7a8e746c: Status 404 returned error can't find the container with id 0241aeae90ccba41107941f724f47941e7a33f7948d75605bfbfb7ce7a8e746c
Oct 30 04:46:15.817831 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:15.817809    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"]
Oct 30 04:46:16.056067 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:16.056027    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4" event={"ID":"f8ff9e15-586c-4b92-9e47-7d79149ebe93","Type":"ContainerStarted","Data":"f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab"}
Oct 30 04:46:16.056067 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:16.056100    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4" event={"ID":"f8ff9e15-586c-4b92-9e47-7d79149ebe93","Type":"ContainerStarted","Data":"4d655e3fc39cdf3c33fb68a90efe8ef2b5f3eb1b9700ca69d01e9b497508fb1d"}
Oct 30 04:46:16.056319 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:16.056279    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"
Oct 30 04:46:16.057631 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:16.057606    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg" event={"ID":"394b0d01-4520-4eb6-8490-719818e3ad43","Type":"ContainerStarted","Data":"0241aeae90ccba41107941f724f47941e7a33f7948d75605bfbfb7ce7a8e746c"}
Oct 30 04:46:16.057745 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:16.057698    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11264-6669e7" podUID="563307c2-9b8f-4666-9f20-09459c7b7b70" containerName="test-curl-11264-6669e7" containerID="cri-o://e255aafb252eaed8241072290539c1cf12f68d2460740bcacf4e17687b7afdc4" gracePeriod=2
Oct 30 04:46:16.100986 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:16.100933    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4" podStartSLOduration=2.10091947 podStartE2EDuration="2.10091947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:16.097219942 +0000 UTC m=+2955.838643039" watchObservedRunningTime="2026-10-30 04:46:16.10091947 +0000 UTC m=+2955.842342566"
Oct 30 04:46:16.111591 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:46:16.111551    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11264-6669e7\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="563307c2-9b8f-4666-9f20-09459c7b7b70" pod="opendatahub/test-curl-11264-6669e7"
Oct 30 04:46:17.066265 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:17.066182    5072 generic.go:356] "Generic (PLEG): container finished" podID="394b0d01-4520-4eb6-8490-719818e3ad43" containerID="501d235cf8ce753991c34db69bcd4c25cce90e358a56c88effe6021d7a004832" exitCode=0
Oct 30 04:46:17.066714 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:17.066264    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg" event={"ID":"394b0d01-4520-4eb6-8490-719818e3ad43","Type":"ContainerDied","Data":"501d235cf8ce753991c34db69bcd4c25cce90e358a56c88effe6021d7a004832"}
Oct 30 04:46:18.251328 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.251004    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:18.292775 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.292746    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394b0d01-4520-4eb6-8490-719818e3ad43-kube-api-access-qqf96\" (UniqueName: \"kubernetes.io/projected/394b0d01-4520-4eb6-8490-719818e3ad43-kube-api-access-qqf96\") pod \"394b0d01-4520-4eb6-8490-719818e3ad43\" (UID: \"394b0d01-4520-4eb6-8490-719818e3ad43\") "
Oct 30 04:46:18.292933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.292848    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-openshift-ca\") pod \"394b0d01-4520-4eb6-8490-719818e3ad43\" (UID: \"394b0d01-4520-4eb6-8490-719818e3ad43\") "
Oct 30 04:46:18.293001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.292944    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-xks-ca\") pod \"394b0d01-4520-4eb6-8490-719818e3ad43\" (UID: \"394b0d01-4520-4eb6-8490-719818e3ad43\") "
Oct 30 04:46:18.293303 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.293274    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-openshift-ca" pod "394b0d01-4520-4eb6-8490-719818e3ad43" (UID: "394b0d01-4520-4eb6-8490-719818e3ad43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:46:18.293422 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.293370    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:18.295215 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.295193    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-xks-ca" pod "394b0d01-4520-4eb6-8490-719818e3ad43" (UID: "394b0d01-4520-4eb6-8490-719818e3ad43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:46:18.295319 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.295235    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394b0d01-4520-4eb6-8490-719818e3ad43-kube-api-access-qqf96" pod "394b0d01-4520-4eb6-8490-719818e3ad43" (UID: "394b0d01-4520-4eb6-8490-719818e3ad43"). InnerVolumeSpecName "kube-api-access-qqf96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:46:18.326923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.326894    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11264-6669e7"
Oct 30 04:46:18.394780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.394742    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/563307c2-9b8f-4666-9f20-09459c7b7b70-kube-api-access-n5hg7\" (UniqueName: \"kubernetes.io/projected/563307c2-9b8f-4666-9f20-09459c7b7b70-kube-api-access-n5hg7\") pod \"563307c2-9b8f-4666-9f20-09459c7b7b70\" (UID: \"563307c2-9b8f-4666-9f20-09459c7b7b70\") "
Oct 30 04:46:18.395251 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.395235    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394b0d01-4520-4eb6-8490-719818e3ad43-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:18.395364 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.395258    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqf96\" (UniqueName: \"kubernetes.io/projected/394b0d01-4520-4eb6-8490-719818e3ad43-kube-api-access-qqf96\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:18.397093 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.397053    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563307c2-9b8f-4666-9f20-09459c7b7b70-kube-api-access-n5hg7" pod "563307c2-9b8f-4666-9f20-09459c7b7b70" (UID: "563307c2-9b8f-4666-9f20-09459c7b7b70"). InnerVolumeSpecName "kube-api-access-n5hg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:46:18.413390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.413361    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="563307c2-9b8f-4666-9f20-09459c7b7b70" path="/var/lib/kubelet/pods/563307c2-9b8f-4666-9f20-09459c7b7b70/volumes"
Oct 30 04:46:18.498235 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:18.498193    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5hg7\" (UniqueName: \"kubernetes.io/projected/563307c2-9b8f-4666-9f20-09459c7b7b70-kube-api-access-n5hg7\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:19.081896 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:19.081734    5072 generic.go:356] "Generic (PLEG): container finished" podID="563307c2-9b8f-4666-9f20-09459c7b7b70" containerID="e255aafb252eaed8241072290539c1cf12f68d2460740bcacf4e17687b7afdc4" exitCode=137
Oct 30 04:46:19.081896 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:19.081795    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11264-6669e7"
Oct 30 04:46:19.081896 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:19.081814    5072 scope.go:122] "RemoveContainer" containerID="e255aafb252eaed8241072290539c1cf12f68d2460740bcacf4e17687b7afdc4"
Oct 30 04:46:19.083889 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:19.083853    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"
Oct 30 04:46:19.084003 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:19.083879    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg" event={"ID":"394b0d01-4520-4eb6-8490-719818e3ad43","Type":"ContainerDied","Data":"0241aeae90ccba41107941f724f47941e7a33f7948d75605bfbfb7ce7a8e746c"}
Oct 30 04:46:19.084003 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:19.083925    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0241aeae90ccba41107941f724f47941e7a33f7948d75605bfbfb7ce7a8e746c"
Oct 30 04:46:19.092574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:19.092541    5072 scope.go:122] "RemoveContainer" containerID="e255aafb252eaed8241072290539c1cf12f68d2460740bcacf4e17687b7afdc4"
Oct 30 04:46:19.092838 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:46:19.092821    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e255aafb252eaed8241072290539c1cf12f68d2460740bcacf4e17687b7afdc4\": container with ID starting with e255aafb252eaed8241072290539c1cf12f68d2460740bcacf4e17687b7afdc4 not found: ID does not exist" containerID="e255aafb252eaed8241072290539c1cf12f68d2460740bcacf4e17687b7afdc4"
Oct 30 04:46:19.092892 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:19.092845    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e255aafb252eaed8241072290539c1cf12f68d2460740bcacf4e17687b7afdc4"} err="failed to get container status \"e255aafb252eaed8241072290539c1cf12f68d2460740bcacf4e17687b7afdc4\": rpc error: code = NotFound desc = could not find container \"e255aafb252eaed8241072290539c1cf12f68d2460740bcacf4e17687b7afdc4\": container with ID starting with e255aafb252eaed8241072290539c1cf12f68d2460740bcacf4e17687b7afdc4 not found: ID does not exist"
Oct 30 04:46:20.006724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:20.006693    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:46:22.022485 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:22.022455    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"
Oct 30 04:46:22.022964 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:22.022535    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"
Oct 30 04:46:23.553991 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.553955    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"]
Oct 30 04:46:23.554980 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.554958    5072 cpu_manager.go:414] "containerMap: removing container" podUID="394b0d01-4520-4eb6-8490-719818e3ad43" containerName="revoke-keys"
Oct 30 04:46:23.554980 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.554982    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394b0d01-4520-4eb6-8490-719818e3ad43" containerName="revoke-keys"
Oct 30 04:46:23.555198 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.555185    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="394b0d01-4520-4eb6-8490-719818e3ad43" containerName="revoke-keys"
Oct 30 04:46:23.560304 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.560283    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"
Oct 30 04:46:23.580564 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.580537    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"]
Oct 30 04:46:23.581874 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.581855    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-91705f-dockercfg-7pgjx\""
Oct 30 04:46:23.588618 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.588592    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-91705f\""
Oct 30 04:46:23.612232 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.612206    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"]
Oct 30 04:46:23.618087 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.618032    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"
Oct 30 04:46:23.637129 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.637101    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"]
Oct 30 04:46:23.664367 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.664334    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xc4z\" (UniqueName: \"kubernetes.io/projected/bf5d06ae-de18-4fef-8538-477be6f43278-kube-api-access-4xc4z\") pod \"payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n\" (UID: \"bf5d06ae-de18-4fef-8538-477be6f43278\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"
Oct 30 04:46:23.664493 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.664465    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d06ae-de18-4fef-8538-477be6f43278-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n\" (UID: \"bf5d06ae-de18-4fef-8538-477be6f43278\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"
Oct 30 04:46:23.664644 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.664616    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96g52\" (UniqueName: \"kubernetes.io/projected/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-kube-api-access-96g52\") pod \"payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb\" (UID: \"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"
Oct 30 04:46:23.664753 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.664734    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb\" (UID: \"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"
Oct 30 04:46:23.766206 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.766172    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96g52\" (UniqueName: \"kubernetes.io/projected/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-kube-api-access-96g52\") pod \"payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb\" (UID: \"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"
Oct 30 04:46:23.766415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.766237    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb\" (UID: \"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"
Oct 30 04:46:23.766415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.766296    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xc4z\" (UniqueName: \"kubernetes.io/projected/bf5d06ae-de18-4fef-8538-477be6f43278-kube-api-access-4xc4z\") pod \"payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n\" (UID: \"bf5d06ae-de18-4fef-8538-477be6f43278\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"
Oct 30 04:46:23.766415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.766368    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d06ae-de18-4fef-8538-477be6f43278-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n\" (UID: \"bf5d06ae-de18-4fef-8538-477be6f43278\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"
Oct 30 04:46:23.767034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.767012    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb\" (UID: \"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"
Oct 30 04:46:23.767141 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.767057    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d06ae-de18-4fef-8538-477be6f43278-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n\" (UID: \"bf5d06ae-de18-4fef-8538-477be6f43278\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"
Oct 30 04:46:23.790651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.790616    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96g52\" (UniqueName: \"kubernetes.io/projected/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-kube-api-access-96g52\") pod \"payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb\" (UID: \"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"
Oct 30 04:46:23.790814 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.790795    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xc4z\" (UniqueName: \"kubernetes.io/projected/bf5d06ae-de18-4fef-8538-477be6f43278-kube-api-access-4xc4z\") pod \"payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n\" (UID: \"bf5d06ae-de18-4fef-8538-477be6f43278\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"
Oct 30 04:46:23.876024 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.875962    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"
Oct 30 04:46:23.935489 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:23.935460    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"
Oct 30 04:46:24.192455 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:46:24.192420    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f01ab3_09fc_4710_b182_a3d2d3dc4f2a.slice/crio-ee79eeb0816af84cb525ee72fcf9815c1ab8e4c4c248b26aa98f6523182fe27f WatchSource:0}: Error finding container ee79eeb0816af84cb525ee72fcf9815c1ab8e4c4c248b26aa98f6523182fe27f: Status 404 returned error can't find the container with id ee79eeb0816af84cb525ee72fcf9815c1ab8e4c4c248b26aa98f6523182fe27f
Oct 30 04:46:24.193949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:24.193925    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"]
Oct 30 04:46:24.258108 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:46:24.258057    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5d06ae_de18_4fef_8538_477be6f43278.slice/crio-c71d3f786bf8dc7827f352304c38ed200af9f73e81e7339918077439b2458da8 WatchSource:0}: Error finding container c71d3f786bf8dc7827f352304c38ed200af9f73e81e7339918077439b2458da8: Status 404 returned error can't find the container with id c71d3f786bf8dc7827f352304c38ed200af9f73e81e7339918077439b2458da8
Oct 30 04:46:24.259457 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:24.259432    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"]
Oct 30 04:46:25.130984 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.130947    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb" event={"ID":"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a","Type":"ContainerStarted","Data":"07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a"}
Oct 30 04:46:25.131528 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.131012    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb" event={"ID":"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a","Type":"ContainerStarted","Data":"ee79eeb0816af84cb525ee72fcf9815c1ab8e4c4c248b26aa98f6523182fe27f"}
Oct 30 04:46:25.131528 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.131055    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"
Oct 30 04:46:25.132844 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.132815    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n" event={"ID":"bf5d06ae-de18-4fef-8538-477be6f43278","Type":"ContainerStarted","Data":"5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b"}
Oct 30 04:46:25.132976 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.132866    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n" event={"ID":"bf5d06ae-de18-4fef-8538-477be6f43278","Type":"ContainerStarted","Data":"c71d3f786bf8dc7827f352304c38ed200af9f73e81e7339918077439b2458da8"}
Oct 30 04:46:25.132976 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.132939    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"
Oct 30 04:46:25.177095 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.177036    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb" podStartSLOduration=2.177022869 podStartE2EDuration="2.177022869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:25.175346151 +0000 UTC m=+2964.916769248" watchObservedRunningTime="2026-10-30 04:46:25.177022869 +0000 UTC m=+2964.918445965"
Oct 30 04:46:25.216878 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.216830    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n" podStartSLOduration=2.21681146 podStartE2EDuration="2.21681146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:25.216700138 +0000 UTC m=+2964.958123237" watchObservedRunningTime="2026-10-30 04:46:25.21681146 +0000 UTC m=+2964.958234554"
Oct 30 04:46:25.601208 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.601173    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"]
Oct 30 04:46:25.605946 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.605925    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:25.624816 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.624787    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"]
Oct 30 04:46:25.626069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.626038    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:46:25.626441 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.626104    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:46:25.688711 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.688675    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6403d038-c61d-4887-affe-adc433266f9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b963057-vltsh\" (UID: \"6403d038-c61d-4887-affe-adc433266f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:25.688864 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.688831    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6403d038-c61d-4887-affe-adc433266f9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b963057-vltsh\" (UID: \"6403d038-c61d-4887-affe-adc433266f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:25.688925 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.688878    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxv6\" (UniqueName: \"kubernetes.io/projected/6403d038-c61d-4887-affe-adc433266f9a-kube-api-access-zrxv6\") pod \"maas-api-revoke-sub-9b963057-vltsh\" (UID: \"6403d038-c61d-4887-affe-adc433266f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:25.789698 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.789666    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6403d038-c61d-4887-affe-adc433266f9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b963057-vltsh\" (UID: \"6403d038-c61d-4887-affe-adc433266f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:25.789889 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.789705    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxv6\" (UniqueName: \"kubernetes.io/projected/6403d038-c61d-4887-affe-adc433266f9a-kube-api-access-zrxv6\") pod \"maas-api-revoke-sub-9b963057-vltsh\" (UID: \"6403d038-c61d-4887-affe-adc433266f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:25.789889 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.789797    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6403d038-c61d-4887-affe-adc433266f9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b963057-vltsh\" (UID: \"6403d038-c61d-4887-affe-adc433266f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:25.790462 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.790440    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6403d038-c61d-4887-affe-adc433266f9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b963057-vltsh\" (UID: \"6403d038-c61d-4887-affe-adc433266f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:25.792306 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.792287    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6403d038-c61d-4887-affe-adc433266f9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b963057-vltsh\" (UID: \"6403d038-c61d-4887-affe-adc433266f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:25.814444 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.814411    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxv6\" (UniqueName: \"kubernetes.io/projected/6403d038-c61d-4887-affe-adc433266f9a-kube-api-access-zrxv6\") pod \"maas-api-revoke-sub-9b963057-vltsh\" (UID: \"6403d038-c61d-4887-affe-adc433266f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:25.923016 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:25.922931    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:26.229671 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:46:26.229643    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6403d038_c61d_4887_affe_adc433266f9a.slice/crio-d27cbcf9b07d443163978d0d27c6ee5dea7fed2b479613b947dc74887c56e2ba WatchSource:0}: Error finding container d27cbcf9b07d443163978d0d27c6ee5dea7fed2b479613b947dc74887c56e2ba: Status 404 returned error can't find the container with id d27cbcf9b07d443163978d0d27c6ee5dea7fed2b479613b947dc74887c56e2ba
Oct 30 04:46:26.231923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:26.231897    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"]
Oct 30 04:46:27.069880 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:27.069853    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"
Oct 30 04:46:27.150735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:27.150696    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh" event={"ID":"6403d038-c61d-4887-affe-adc433266f9a","Type":"ContainerStarted","Data":"614da05a521d035c6a9adc734d70c533c1c1e54a34c713d56c1ef01de397b5e3"}
Oct 30 04:46:27.150946 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:27.150752    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh" event={"ID":"6403d038-c61d-4887-affe-adc433266f9a","Type":"ContainerStarted","Data":"d27cbcf9b07d443163978d0d27c6ee5dea7fed2b479613b947dc74887c56e2ba"}
Oct 30 04:46:27.194481 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:27.194430    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh" podStartSLOduration=2.194415995 podStartE2EDuration="2.194415995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:27.193165761 +0000 UTC m=+2966.934588869" watchObservedRunningTime="2026-10-30 04:46:27.194415995 +0000 UTC m=+2966.935839094"
Oct 30 04:46:28.159300 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:28.159266    5072 generic.go:356] "Generic (PLEG): container finished" podID="6403d038-c61d-4887-affe-adc433266f9a" containerID="614da05a521d035c6a9adc734d70c533c1c1e54a34c713d56c1ef01de397b5e3" exitCode=0
Oct 30 04:46:28.159766 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:28.159357    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh" event={"ID":"6403d038-c61d-4887-affe-adc433266f9a","Type":"ContainerDied","Data":"614da05a521d035c6a9adc734d70c533c1c1e54a34c713d56c1ef01de397b5e3"}
Oct 30 04:46:29.317675 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:29.317649    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:29.438418 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:29.438388    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6403d038-c61d-4887-affe-adc433266f9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6403d038-c61d-4887-affe-adc433266f9a-maas-api-xks-ca\") pod \"6403d038-c61d-4887-affe-adc433266f9a\" (UID: \"6403d038-c61d-4887-affe-adc433266f9a\") "
Oct 30 04:46:29.438601 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:29.438588    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6403d038-c61d-4887-affe-adc433266f9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6403d038-c61d-4887-affe-adc433266f9a-maas-api-openshift-ca\") pod \"6403d038-c61d-4887-affe-adc433266f9a\" (UID: \"6403d038-c61d-4887-affe-adc433266f9a\") "
Oct 30 04:46:29.438662 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:29.438612    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6403d038-c61d-4887-affe-adc433266f9a-kube-api-access-zrxv6\" (UniqueName: \"kubernetes.io/projected/6403d038-c61d-4887-affe-adc433266f9a-kube-api-access-zrxv6\") pod \"6403d038-c61d-4887-affe-adc433266f9a\" (UID: \"6403d038-c61d-4887-affe-adc433266f9a\") "
Oct 30 04:46:29.438943 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:29.438916    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6403d038-c61d-4887-affe-adc433266f9a-maas-api-openshift-ca" pod "6403d038-c61d-4887-affe-adc433266f9a" (UID: "6403d038-c61d-4887-affe-adc433266f9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:46:29.440788 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:29.440759    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6403d038-c61d-4887-affe-adc433266f9a-maas-api-xks-ca" pod "6403d038-c61d-4887-affe-adc433266f9a" (UID: "6403d038-c61d-4887-affe-adc433266f9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:46:29.440880 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:29.440836    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6403d038-c61d-4887-affe-adc433266f9a-kube-api-access-zrxv6" pod "6403d038-c61d-4887-affe-adc433266f9a" (UID: "6403d038-c61d-4887-affe-adc433266f9a"). InnerVolumeSpecName "kube-api-access-zrxv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:46:29.540157 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:29.540119    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6403d038-c61d-4887-affe-adc433266f9a-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:29.540157 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:29.540155    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxv6\" (UniqueName: \"kubernetes.io/projected/6403d038-c61d-4887-affe-adc433266f9a-kube-api-access-zrxv6\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:29.540157 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:29.540171    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6403d038-c61d-4887-affe-adc433266f9a-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:30.175529 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:30.175502    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"
Oct 30 04:46:30.175696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:30.175537    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh" event={"ID":"6403d038-c61d-4887-affe-adc433266f9a","Type":"ContainerDied","Data":"d27cbcf9b07d443163978d0d27c6ee5dea7fed2b479613b947dc74887c56e2ba"}
Oct 30 04:46:30.175696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:30.175586    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d27cbcf9b07d443163978d0d27c6ee5dea7fed2b479613b947dc74887c56e2ba"
Oct 30 04:46:36.143164 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:36.143134    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"
Oct 30 04:46:36.143701 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:36.143552    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"
Oct 30 04:46:37.777449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:37.777416    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"]
Oct 30 04:46:41.759455 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:46:41.759385    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:46:47.395241 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:47.395209    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"]
Oct 30 04:46:52.803222 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:52.803164    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5" podUID="0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c" containerName="main" containerID="cri-o://fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880" gracePeriod=45
Oct 30 04:46:53.088493 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.088465    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:46:53.222307 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.222276    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-dshm\") pod \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") "
Oct 30 04:46:53.222565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.222354    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tmp-dir\") pod \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") "
Oct 30 04:46:53.222565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.222374    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tls-certs\") pod \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") "
Oct 30 04:46:53.222565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.222404    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-home\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-home\") pod \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") "
Oct 30 04:46:53.222565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.222488    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-kube-api-access-r9shn\" (UniqueName: \"kubernetes.io/projected/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-kube-api-access-r9shn\") pod \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") "
Oct 30 04:46:53.222565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.222541    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-model-cache\") pod \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\" (UID: \"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c\") "
Oct 30 04:46:53.223256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.222902    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-home" pod "0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c" (UID: "0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:46:53.223256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.222909    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-model-cache" pod "0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c" (UID: "0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:46:53.223256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.223176    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:53.223256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.223197    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:53.223496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.223472    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tmp-dir" pod "0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c" (UID: "0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:46:53.225847 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.225826    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-dshm" pod "0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c" (UID: "0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:46:53.225940 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.225853    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tls-certs" pod "0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c" (UID: "0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:46:53.225940 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.225859    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-kube-api-access-r9shn" pod "0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c" (UID: "0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c"). InnerVolumeSpecName "kube-api-access-r9shn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:46:53.324525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.324465    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:53.324525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.324489    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:53.324525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.324497    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:53.324525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.324507    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9shn\" (UniqueName: \"kubernetes.io/projected/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c-kube-api-access-r9shn\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:46:53.350293 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.350265    5072 generic.go:356] "Generic (PLEG): container finished" podID="0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c" containerID="fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880" exitCode=0
Oct 30 04:46:53.350397 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.350334    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"
Oct 30 04:46:53.350397 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.350359    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5" event={"ID":"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c","Type":"ContainerDied","Data":"fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880"}
Oct 30 04:46:53.350490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.350411    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5" event={"ID":"0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c","Type":"ContainerDied","Data":"0eefe5a972b511013639768b8dc5f3e0118c3307ee22c61cc701746a90aff5e7"}
Oct 30 04:46:53.350490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.350434    5072 scope.go:122] "RemoveContainer" containerID="fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880"
Oct 30 04:46:53.363377 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.363359    5072 scope.go:122] "RemoveContainer" containerID="fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880"
Oct 30 04:46:53.363684 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:46:53.363630    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880\": container with ID starting with fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880 not found: ID does not exist" containerID="fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880"
Oct 30 04:46:53.363684 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.363662    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880"} err="failed to get container status \"fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880\": rpc error: code = NotFound desc = could not find container \"fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880\": container with ID starting with fdfb619b16aa0f6234e15b9abc95eaa56ac91d79b1e3a0d9f59ae18bb4406880 not found: ID does not exist"
Oct 30 04:46:53.405328 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.405301    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"]
Oct 30 04:46:53.419599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:53.419572    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4c2415/auth-test-model-4c2415-kserve-748fb5cf57-jr4q5"]
Oct 30 04:46:54.410618 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.410584    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c" path="/var/lib/kubelet/pods/0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c/volumes"
Oct 30 04:46:54.800742 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.800707    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"]
Oct 30 04:46:54.801415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.801400    5072 cpu_manager.go:414] "containerMap: removing container" podUID="6403d038-c61d-4887-affe-adc433266f9a" containerName="revoke-keys"
Oct 30 04:46:54.801490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.801417    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6403d038-c61d-4887-affe-adc433266f9a" containerName="revoke-keys"
Oct 30 04:46:54.801490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.801476    5072 cpu_manager.go:414] "containerMap: removing container" podUID="0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c" containerName="main"
Oct 30 04:46:54.801490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.801481    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c" containerName="main"
Oct 30 04:46:54.801615 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.801574    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b01d3a7-44b9-4b1c-b6a6-f3ed540e2c9c" containerName="main"
Oct 30 04:46:54.801615 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.801584    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6403d038-c61d-4887-affe-adc433266f9a" containerName="revoke-keys"
Oct 30 04:46:54.806696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.806677    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"
Oct 30 04:46:54.825342 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.825320    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"]
Oct 30 04:46:54.834380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.834042    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-721c85\""
Oct 30 04:46:54.834380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.834134    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-721c85-dockercfg-52qn7\""
Oct 30 04:46:54.902257 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.902215    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"]
Oct 30 04:46:54.914468 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.914441    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"
Oct 30 04:46:54.920732 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.920261    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"]
Oct 30 04:46:54.949346 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.949147    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc684e7d-f6c1-4011-8632-810ba45a1970-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr\" (UID: \"bc684e7d-f6c1-4011-8632-810ba45a1970\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"
Oct 30 04:46:54.949346 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:54.949215    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdxr\" (UniqueName: \"kubernetes.io/projected/bc684e7d-f6c1-4011-8632-810ba45a1970-kube-api-access-6vdxr\") pod \"payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr\" (UID: \"bc684e7d-f6c1-4011-8632-810ba45a1970\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"
Oct 30 04:46:55.050356 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.050319    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc684e7d-f6c1-4011-8632-810ba45a1970-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr\" (UID: \"bc684e7d-f6c1-4011-8632-810ba45a1970\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"
Oct 30 04:46:55.050565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.050376    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faae7ed3-be0f-445a-9b4e-fe7674637de2-plugins-config-volume\") pod \"payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq\" (UID: \"faae7ed3-be0f-445a-9b4e-fe7674637de2\") " pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"
Oct 30 04:46:55.050565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.050438    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdxr\" (UniqueName: \"kubernetes.io/projected/bc684e7d-f6c1-4011-8632-810ba45a1970-kube-api-access-6vdxr\") pod \"payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr\" (UID: \"bc684e7d-f6c1-4011-8632-810ba45a1970\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"
Oct 30 04:46:55.050565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.050514    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lc7q\" (UniqueName: \"kubernetes.io/projected/faae7ed3-be0f-445a-9b4e-fe7674637de2-kube-api-access-2lc7q\") pod \"payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq\" (UID: \"faae7ed3-be0f-445a-9b4e-fe7674637de2\") " pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"
Oct 30 04:46:55.051052 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.051008    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc684e7d-f6c1-4011-8632-810ba45a1970-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr\" (UID: \"bc684e7d-f6c1-4011-8632-810ba45a1970\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"
Oct 30 04:46:55.087055 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.087022    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdxr\" (UniqueName: \"kubernetes.io/projected/bc684e7d-f6c1-4011-8632-810ba45a1970-kube-api-access-6vdxr\") pod \"payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr\" (UID: \"bc684e7d-f6c1-4011-8632-810ba45a1970\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"
Oct 30 04:46:55.120955 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.120923    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"
Oct 30 04:46:55.151811 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.151778    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faae7ed3-be0f-445a-9b4e-fe7674637de2-plugins-config-volume\") pod \"payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq\" (UID: \"faae7ed3-be0f-445a-9b4e-fe7674637de2\") " pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"
Oct 30 04:46:55.151961 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.151849    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lc7q\" (UniqueName: \"kubernetes.io/projected/faae7ed3-be0f-445a-9b4e-fe7674637de2-kube-api-access-2lc7q\") pod \"payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq\" (UID: \"faae7ed3-be0f-445a-9b4e-fe7674637de2\") " pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"
Oct 30 04:46:55.152452 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.152434    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faae7ed3-be0f-445a-9b4e-fe7674637de2-plugins-config-volume\") pod \"payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq\" (UID: \"faae7ed3-be0f-445a-9b4e-fe7674637de2\") " pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"
Oct 30 04:46:55.193211 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.193175    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lc7q\" (UniqueName: \"kubernetes.io/projected/faae7ed3-be0f-445a-9b4e-fe7674637de2-kube-api-access-2lc7q\") pod \"payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq\" (UID: \"faae7ed3-be0f-445a-9b4e-fe7674637de2\") " pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"
Oct 30 04:46:55.292735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.292507    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"
Oct 30 04:46:55.504130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.503744    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"]
Oct 30 04:46:55.640583 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:46:55.640555    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaae7ed3_be0f_445a_9b4e_fe7674637de2.slice/crio-750b15f2d673824b92d536feee0e770e3e6033b489a7bf0d2be29ae9d42ae3c1 WatchSource:0}: Error finding container 750b15f2d673824b92d536feee0e770e3e6033b489a7bf0d2be29ae9d42ae3c1: Status 404 returned error can't find the container with id 750b15f2d673824b92d536feee0e770e3e6033b489a7bf0d2be29ae9d42ae3c1
Oct 30 04:46:55.642848 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:55.642824    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"]
Oct 30 04:46:56.412539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:56.412504    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr" event={"ID":"bc684e7d-f6c1-4011-8632-810ba45a1970","Type":"ContainerStarted","Data":"744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43"}
Oct 30 04:46:56.412539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:56.412547    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr" event={"ID":"bc684e7d-f6c1-4011-8632-810ba45a1970","Type":"ContainerStarted","Data":"2ac9fe9af645f894c7ce56a37d67098970f01cb632b23644243d7bd4c6518c79"}
Oct 30 04:46:56.412811 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:56.412566    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"
Oct 30 04:46:56.414285 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:56.414254    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq" event={"ID":"faae7ed3-be0f-445a-9b4e-fe7674637de2","Type":"ContainerStarted","Data":"0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96"}
Oct 30 04:46:56.414402 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:56.414307    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq" event={"ID":"faae7ed3-be0f-445a-9b4e-fe7674637de2","Type":"ContainerStarted","Data":"750b15f2d673824b92d536feee0e770e3e6033b489a7bf0d2be29ae9d42ae3c1"}
Oct 30 04:46:56.414402 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:56.414339    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"
Oct 30 04:46:56.464100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:56.455969    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr" podStartSLOduration=2.45595135 podStartE2EDuration="2.45595135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:56.453434156 +0000 UTC m=+2996.194857264" watchObservedRunningTime="2026-10-30 04:46:56.45595135 +0000 UTC m=+2996.197374448"
Oct 30 04:46:56.511504 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:46:56.511444    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq" podStartSLOduration=2.511428369 podStartE2EDuration="2.511428369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:56.510290238 +0000 UTC m=+2996.251713336" watchObservedRunningTime="2026-10-30 04:46:56.511428369 +0000 UTC m=+2996.252851467"
Oct 30 04:47:00.811997 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:00.811882    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:47:00.835429 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:00.826466    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:47:02.418612 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.418543    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b" podUID="b837af6e-7029-42e9-85a0-14edfa6fbccd" containerName="main" containerID="cri-o://fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9" gracePeriod=45
Oct 30 04:47:02.519416 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.519377    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b" podUID="b837af6e-7029-42e9-85a0-14edfa6fbccd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": dial tcp 10.132.0.117:8000: connect: connection refused"
Oct 30 04:47:02.705311 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.705281    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:47:02.884180 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.884151    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-dshm\") pod \"b837af6e-7029-42e9-85a0-14edfa6fbccd\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") "
Oct 30 04:47:02.884373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.884299    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-tmp-dir\") pod \"b837af6e-7029-42e9-85a0-14edfa6fbccd\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") "
Oct 30 04:47:02.884373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.884334    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b837af6e-7029-42e9-85a0-14edfa6fbccd-kube-api-access-dghfg\" (UniqueName: \"kubernetes.io/projected/b837af6e-7029-42e9-85a0-14edfa6fbccd-kube-api-access-dghfg\") pod \"b837af6e-7029-42e9-85a0-14edfa6fbccd\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") "
Oct 30 04:47:02.884373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.884366    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-model-cache\") pod \"b837af6e-7029-42e9-85a0-14edfa6fbccd\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") "
Oct 30 04:47:02.884549 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.884462    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b837af6e-7029-42e9-85a0-14edfa6fbccd-tls-certs\" (UniqueName: \"kubernetes.io/secret/b837af6e-7029-42e9-85a0-14edfa6fbccd-tls-certs\") pod \"b837af6e-7029-42e9-85a0-14edfa6fbccd\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") "
Oct 30 04:47:02.884610 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.884550    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-home\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-home\") pod \"b837af6e-7029-42e9-85a0-14edfa6fbccd\" (UID: \"b837af6e-7029-42e9-85a0-14edfa6fbccd\") "
Oct 30 04:47:02.884667 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.884567    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-tmp-dir" pod "b837af6e-7029-42e9-85a0-14edfa6fbccd" (UID: "b837af6e-7029-42e9-85a0-14edfa6fbccd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:47:02.884667 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.884645    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-model-cache" pod "b837af6e-7029-42e9-85a0-14edfa6fbccd" (UID: "b837af6e-7029-42e9-85a0-14edfa6fbccd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:47:02.884812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.884785    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-home" pod "b837af6e-7029-42e9-85a0-14edfa6fbccd" (UID: "b837af6e-7029-42e9-85a0-14edfa6fbccd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:47:02.885165 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.885061    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:02.885165 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.885107    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:02.885165 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.885121    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:02.886724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.886698    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-dshm" pod "b837af6e-7029-42e9-85a0-14edfa6fbccd" (UID: "b837af6e-7029-42e9-85a0-14edfa6fbccd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:47:02.886831 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.886769    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b837af6e-7029-42e9-85a0-14edfa6fbccd-kube-api-access-dghfg" pod "b837af6e-7029-42e9-85a0-14edfa6fbccd" (UID: "b837af6e-7029-42e9-85a0-14edfa6fbccd"). InnerVolumeSpecName "kube-api-access-dghfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:47:02.887008 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.886991    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b837af6e-7029-42e9-85a0-14edfa6fbccd-tls-certs" pod "b837af6e-7029-42e9-85a0-14edfa6fbccd" (UID: "b837af6e-7029-42e9-85a0-14edfa6fbccd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:47:02.986518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.986488    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b837af6e-7029-42e9-85a0-14edfa6fbccd-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:02.986518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.986513    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b837af6e-7029-42e9-85a0-14edfa6fbccd-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:02.986518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:02.986523    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dghfg\" (UniqueName: \"kubernetes.io/projected/b837af6e-7029-42e9-85a0-14edfa6fbccd-kube-api-access-dghfg\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:03.468956 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:03.468914    5072 generic.go:356] "Generic (PLEG): container finished" podID="b837af6e-7029-42e9-85a0-14edfa6fbccd" containerID="fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9" exitCode=0
Oct 30 04:47:03.468956 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:03.468955    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"
Oct 30 04:47:03.469513 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:03.468991    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b" event={"ID":"b837af6e-7029-42e9-85a0-14edfa6fbccd","Type":"ContainerDied","Data":"fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9"}
Oct 30 04:47:03.469513 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:03.469035    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b" event={"ID":"b837af6e-7029-42e9-85a0-14edfa6fbccd","Type":"ContainerDied","Data":"2cbded1ee1aa98c4ea666c658cf3a2b24a0fe9a1a04d5fa8464c64750852fe82"}
Oct 30 04:47:03.469513 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:03.469052    5072 scope.go:122] "RemoveContainer" containerID="fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9"
Oct 30 04:47:03.481376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:03.481357    5072 scope.go:122] "RemoveContainer" containerID="fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9"
Oct 30 04:47:03.481663 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:47:03.481646    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9\": container with ID starting with fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9 not found: ID does not exist" containerID="fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9"
Oct 30 04:47:03.481726 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:03.481669    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9"} err="failed to get container status \"fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9\": rpc error: code = NotFound desc = could not find container \"fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9\": container with ID starting with fc538015d205188ce169e5257c8f6c321a321784ce10e8282fa50fba4fa409f9 not found: ID does not exist"
Oct 30 04:47:03.523742 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:03.523708    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"]
Oct 30 04:47:03.536629 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:03.536605    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bbce72/auth-test-model-bbce72-kserve-58fc946fb9-7f46b"]
Oct 30 04:47:04.409461 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:04.409431    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b837af6e-7029-42e9-85a0-14edfa6fbccd" path="/var/lib/kubelet/pods/b837af6e-7029-42e9-85a0-14edfa6fbccd/volumes"
Oct 30 04:47:04.478867 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:04.478840    5072 generic.go:356] "Generic (PLEG): container finished" podID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerID="84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0" exitCode=0
Oct 30 04:47:04.478867 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:04.478871    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" event={"ID":"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9","Type":"ContainerDied","Data":"84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0"}
Oct 30 04:47:04.479306 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:04.479245    5072 scope.go:122] "RemoveContainer" containerID="84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0"
Oct 30 04:47:04.480970 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:04.480943    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:47:04.481047 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:04.481014    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:47:04.481113 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:04.481045    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:47:05.491496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:05.491465    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" event={"ID":"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9","Type":"ContainerStarted","Data":"8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648"}
Oct 30 04:47:05.972919 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:05.972881    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:47:06.202524 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:06.202491    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:47:06.203792 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:06.203766    5072 patch_prober.go:29] interesting pod/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 04:47:06.203924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:06.203815    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 04:47:07.202960 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:07.202926    5072 patch_prober.go:29] interesting pod/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 04:47:07.203517 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:07.202984    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 04:47:07.424702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:07.424672    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"
Oct 30 04:47:07.424874 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:07.424742    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"
Oct 30 04:47:08.202454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:08.202414    5072 patch_prober.go:29] interesting pod/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 04:47:08.202664 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:08.202482    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 04:47:09.202192 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:09.202156    5072 patch_prober.go:29] interesting pod/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 04:47:09.202697 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:09.202246    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 04:47:10.202434 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:10.202401    5072 patch_prober.go:29] interesting pod/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 04:47:10.202827 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:10.202457    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 04:47:11.207872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:11.207839    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:47:11.209158 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:11.209136    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 04:47:11.751587 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:47:11.751554    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:47:23.751059 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.751029    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"]
Oct 30 04:47:23.751813 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.751797    5072 cpu_manager.go:414] "containerMap: removing container" podUID="b837af6e-7029-42e9-85a0-14edfa6fbccd" containerName="main"
Oct 30 04:47:23.751858 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.751814    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b837af6e-7029-42e9-85a0-14edfa6fbccd" containerName="main"
Oct 30 04:47:23.751931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.751922    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b837af6e-7029-42e9-85a0-14edfa6fbccd" containerName="main"
Oct 30 04:47:23.759207 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.759182    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:23.775125 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.775094    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"]
Oct 30 04:47:23.782429 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.782402    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:47:23.782560 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.782522    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:47:23.850646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.850614    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v77sf\" (UniqueName: \"kubernetes.io/projected/004ef81e-66c7-43df-8891-9d2b45e431e3-kube-api-access-v77sf\") pod \"maas-api-revoke-sub-5a747ef2-lqf5k\" (UID: \"004ef81e-66c7-43df-8891-9d2b45e431e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:23.850785 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.850735    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a747ef2-lqf5k\" (UID: \"004ef81e-66c7-43df-8891-9d2b45e431e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:23.850829 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.850786    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a747ef2-lqf5k\" (UID: \"004ef81e-66c7-43df-8891-9d2b45e431e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:23.952435 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.952399    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77sf\" (UniqueName: \"kubernetes.io/projected/004ef81e-66c7-43df-8891-9d2b45e431e3-kube-api-access-v77sf\") pod \"maas-api-revoke-sub-5a747ef2-lqf5k\" (UID: \"004ef81e-66c7-43df-8891-9d2b45e431e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:23.952557 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.952493    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a747ef2-lqf5k\" (UID: \"004ef81e-66c7-43df-8891-9d2b45e431e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:23.952557 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.952534    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a747ef2-lqf5k\" (UID: \"004ef81e-66c7-43df-8891-9d2b45e431e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:23.953293 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.953276    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a747ef2-lqf5k\" (UID: \"004ef81e-66c7-43df-8891-9d2b45e431e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:23.955064 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.955047    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a747ef2-lqf5k\" (UID: \"004ef81e-66c7-43df-8891-9d2b45e431e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:23.976573 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:23.976548    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v77sf\" (UniqueName: \"kubernetes.io/projected/004ef81e-66c7-43df-8891-9d2b45e431e3-kube-api-access-v77sf\") pod \"maas-api-revoke-sub-5a747ef2-lqf5k\" (UID: \"004ef81e-66c7-43df-8891-9d2b45e431e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:24.078505 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:24.078440    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:24.381981 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:47:24.381945    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004ef81e_66c7_43df_8891_9d2b45e431e3.slice/crio-047a020f28e907893e6ad7cd0e923c79ef4f5a6fd406c136ef5382dc11156c19 WatchSource:0}: Error finding container 047a020f28e907893e6ad7cd0e923c79ef4f5a6fd406c136ef5382dc11156c19: Status 404 returned error can't find the container with id 047a020f28e907893e6ad7cd0e923c79ef4f5a6fd406c136ef5382dc11156c19
Oct 30 04:47:24.383237 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:24.383213    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"]
Oct 30 04:47:24.641045 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:24.640963    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k" event={"ID":"004ef81e-66c7-43df-8891-9d2b45e431e3","Type":"ContainerStarted","Data":"047a020f28e907893e6ad7cd0e923c79ef4f5a6fd406c136ef5382dc11156c19"}
Oct 30 04:47:25.649753 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:25.649673    5072 generic.go:356] "Generic (PLEG): container finished" podID="004ef81e-66c7-43df-8891-9d2b45e431e3" containerID="8a90b469c09242d9650c9e9e12390541e7b7805109a89ad58f77b5d07f4234a2" exitCode=0
Oct 30 04:47:25.650199 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:25.649822    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k" event={"ID":"004ef81e-66c7-43df-8891-9d2b45e431e3","Type":"ContainerDied","Data":"8a90b469c09242d9650c9e9e12390541e7b7805109a89ad58f77b5d07f4234a2"}
Oct 30 04:47:26.845505 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:26.845470    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:26.990525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:26.990486    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-xks-ca\") pod \"004ef81e-66c7-43df-8891-9d2b45e431e3\" (UID: \"004ef81e-66c7-43df-8891-9d2b45e431e3\") "
Oct 30 04:47:26.990749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:26.990595    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-openshift-ca\") pod \"004ef81e-66c7-43df-8891-9d2b45e431e3\" (UID: \"004ef81e-66c7-43df-8891-9d2b45e431e3\") "
Oct 30 04:47:26.990749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:26.990677    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004ef81e-66c7-43df-8891-9d2b45e431e3-kube-api-access-v77sf\" (UniqueName: \"kubernetes.io/projected/004ef81e-66c7-43df-8891-9d2b45e431e3-kube-api-access-v77sf\") pod \"004ef81e-66c7-43df-8891-9d2b45e431e3\" (UID: \"004ef81e-66c7-43df-8891-9d2b45e431e3\") "
Oct 30 04:47:26.991433 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:26.991386    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-openshift-ca" pod "004ef81e-66c7-43df-8891-9d2b45e431e3" (UID: "004ef81e-66c7-43df-8891-9d2b45e431e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:47:26.995556 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:26.995527    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004ef81e-66c7-43df-8891-9d2b45e431e3-kube-api-access-v77sf" pod "004ef81e-66c7-43df-8891-9d2b45e431e3" (UID: "004ef81e-66c7-43df-8891-9d2b45e431e3"). InnerVolumeSpecName "kube-api-access-v77sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:47:27.011101 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:27.008232    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-xks-ca" pod "004ef81e-66c7-43df-8891-9d2b45e431e3" (UID: "004ef81e-66c7-43df-8891-9d2b45e431e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:47:27.092027 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:27.091994    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:27.092027 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:27.092020    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v77sf\" (UniqueName: \"kubernetes.io/projected/004ef81e-66c7-43df-8891-9d2b45e431e3-kube-api-access-v77sf\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:27.092027 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:27.092030    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004ef81e-66c7-43df-8891-9d2b45e431e3-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:27.669807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:27.669771    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k" event={"ID":"004ef81e-66c7-43df-8891-9d2b45e431e3","Type":"ContainerDied","Data":"047a020f28e907893e6ad7cd0e923c79ef4f5a6fd406c136ef5382dc11156c19"}
Oct 30 04:47:27.669966 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:27.669843    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="047a020f28e907893e6ad7cd0e923c79ef4f5a6fd406c136ef5382dc11156c19"
Oct 30 04:47:27.669966 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:27.669786    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"
Oct 30 04:47:29.003685 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.003649    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"]
Oct 30 04:47:29.004432 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.004408    5072 cpu_manager.go:414] "containerMap: removing container" podUID="004ef81e-66c7-43df-8891-9d2b45e431e3" containerName="revoke-keys"
Oct 30 04:47:29.004432 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.004432    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004ef81e-66c7-43df-8891-9d2b45e431e3" containerName="revoke-keys"
Oct 30 04:47:29.004553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.004532    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="004ef81e-66c7-43df-8891-9d2b45e431e3" containerName="revoke-keys"
Oct 30 04:47:29.008374 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.008351    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.025849 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.025820    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"]
Oct 30 04:47:29.034958 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.034927    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"agc-cred-259031349671\""
Oct 30 04:47:29.035113 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.035022    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"payload-processing-external-model-plugins-e2e-extmod-b-503804\""
Oct 30 04:47:29.035113 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.035035    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"openshift-service-ca.crt\""
Oct 30 04:47:29.035243 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.035181    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"payload-processing-external-model-e2e--0b0ba164-dockercfg-xpm8t\""
Oct 30 04:47:29.036432 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.036388    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"routing-overlay\""
Oct 30 04:47:29.036432 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.036407    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"kube-root-ca.crt\""
Oct 30 04:47:29.116169 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.116134    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvf7\" (UniqueName: \"kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-kube-api-access-xgvf7\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.116407 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.116388    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.116483 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.116432    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.116483 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.116452    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.217473 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.217434    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.217669 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.217481    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.217669 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.217510    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.217669 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.217542    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvf7\" (UniqueName: \"kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-kube-api-access-xgvf7\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.218468 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.218438    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.218676 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.218467    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.223644 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.223614    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.252659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.252631    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvf7\" (UniqueName: \"kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-kube-api-access-xgvf7\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.323062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.322922    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:29.652659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.652628    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"]
Oct 30 04:47:29.653415 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:47:29.653386    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700b211e_c846_4d46_96eb_b2ef28e93c0e.slice/crio-4b02c9134e4becd5f819ca5c784ddeb7f0950a5210bdf22cc69b9ab1354f8acb WatchSource:0}: Error finding container 4b02c9134e4becd5f819ca5c784ddeb7f0950a5210bdf22cc69b9ab1354f8acb: Status 404 returned error can't find the container with id 4b02c9134e4becd5f819ca5c784ddeb7f0950a5210bdf22cc69b9ab1354f8acb
Oct 30 04:47:29.685446 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:29.685416    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6" event={"ID":"700b211e-c846-4d46-96eb-b2ef28e93c0e","Type":"ContainerStarted","Data":"4b02c9134e4becd5f819ca5c784ddeb7f0950a5210bdf22cc69b9ab1354f8acb"}
Oct 30 04:47:30.694021 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:30.693979    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6" event={"ID":"700b211e-c846-4d46-96eb-b2ef28e93c0e","Type":"ContainerStarted","Data":"b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256"}
Oct 30 04:47:30.694478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:30.694187    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:30.736643 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:30.736594    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6" podStartSLOduration=2.736582224 podStartE2EDuration="2.736582224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:47:30.736401793 +0000 UTC m=+3030.477824890" watchObservedRunningTime="2026-10-30 04:47:30.736582224 +0000 UTC m=+3030.478005321"
Oct 30 04:47:34.176243 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.176209    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"]
Oct 30 04:47:34.180700 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.180681    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.196385 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.196362    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"agc-cred-2c8109ac218b\""
Oct 30 04:47:34.200941 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.200916    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"]
Oct 30 04:47:34.208536 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.208308    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"payload-processing-external-model-e2e--30ae2d69-dockercfg-n79gr\""
Oct 30 04:47:34.208536 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.208448    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b195d0\""
Oct 30 04:47:34.208536 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.208512    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"openshift-service-ca.crt\""
Oct 30 04:47:34.209308 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.209275    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"routing-overlay\""
Oct 30 04:47:34.209418 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.209334    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"kube-root-ca.crt\""
Oct 30 04:47:34.276447 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.276406    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.276447 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.276452    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.276696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.276660    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.276696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.276692    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkrvj\" (UniqueName: \"kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-kube-api-access-zkrvj\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.377431 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.377395    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.377665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.377446    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.377752 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.377726    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.377817 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.377771    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrvj\" (UniqueName: \"kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-kube-api-access-zkrvj\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.378356 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.378321    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.378720 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.378702    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.380442 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.380411    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.402776 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.402750    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkrvj\" (UniqueName: \"kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-kube-api-access-zkrvj\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.496051 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.496017    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:34.812873 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:47:34.812837    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca39491_d862_43a9_a037_1f8b994c91a9.slice/crio-2b1725ec7e05e891f047f932eceab53374402284928dc62afff10b1fbfdbbc22 WatchSource:0}: Error finding container 2b1725ec7e05e891f047f932eceab53374402284928dc62afff10b1fbfdbbc22: Status 404 returned error can't find the container with id 2b1725ec7e05e891f047f932eceab53374402284928dc62afff10b1fbfdbbc22
Oct 30 04:47:34.813563 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:34.813535    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"]
Oct 30 04:47:35.153336 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:35.153253    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"]
Oct 30 04:47:35.741516 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:35.741478    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj" event={"ID":"9ca39491-d862-43a9-a037-1f8b994c91a9","Type":"ContainerStarted","Data":"1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831"}
Oct 30 04:47:35.741945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:35.741546    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj" event={"ID":"9ca39491-d862-43a9-a037-1f8b994c91a9","Type":"ContainerStarted","Data":"2b1725ec7e05e891f047f932eceab53374402284928dc62afff10b1fbfdbbc22"}
Oct 30 04:47:35.741945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:35.741609    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:35.786263 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:35.786214    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj" podStartSLOduration=1.786199699 podStartE2EDuration="1.786199699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:47:35.782980524 +0000 UTC m=+3035.524403623" watchObservedRunningTime="2026-10-30 04:47:35.786199699 +0000 UTC m=+3035.527622796"
Oct 30 04:47:40.154207 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.154146    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7" podUID="bd3a1d2a-bd77-4141-a4c2-ad672da165ee" containerName="payload-processing" containerID="cri-o://3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132" gracePeriod=25
Oct 30 04:47:40.453787 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.453756    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:47:40.560930 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.560894    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-plugins-config-volume\") pod \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") "
Oct 30 04:47:40.561130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.560949    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-kube-api-access-hvjd7\" (UniqueName: \"kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-kube-api-access-hvjd7\") pod \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") "
Oct 30 04:47:40.561130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.560983    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-provider-credentials\") pod \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") "
Oct 30 04:47:40.561130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.561003    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-routing-overlay-volume\") pod \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\" (UID: \"bd3a1d2a-bd77-4141-a4c2-ad672da165ee\") "
Oct 30 04:47:40.561397 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.561372    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-plugins-config-volume" pod "bd3a1d2a-bd77-4141-a4c2-ad672da165ee" (UID: "bd3a1d2a-bd77-4141-a4c2-ad672da165ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:47:40.561460 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.561435    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-routing-overlay-volume" pod "bd3a1d2a-bd77-4141-a4c2-ad672da165ee" (UID: "bd3a1d2a-bd77-4141-a4c2-ad672da165ee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:47:40.561556 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.561542    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:40.561599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.561560    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:40.563518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.563499    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-provider-credentials" pod "bd3a1d2a-bd77-4141-a4c2-ad672da165ee" (UID: "bd3a1d2a-bd77-4141-a4c2-ad672da165ee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:47:40.563719 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.563704    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-kube-api-access-hvjd7" pod "bd3a1d2a-bd77-4141-a4c2-ad672da165ee" (UID: "bd3a1d2a-bd77-4141-a4c2-ad672da165ee"). InnerVolumeSpecName "kube-api-access-hvjd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:47:40.662342 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.662319    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvjd7\" (UniqueName: \"kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-kube-api-access-hvjd7\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:40.662342 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.662342    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd3a1d2a-bd77-4141-a4c2-ad672da165ee-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:47:40.778474 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.778437    5072 generic.go:356] "Generic (PLEG): container finished" podID="bd3a1d2a-bd77-4141-a4c2-ad672da165ee" containerID="3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132" exitCode=0
Oct 30 04:47:40.778652 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.778471    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7" event={"ID":"bd3a1d2a-bd77-4141-a4c2-ad672da165ee","Type":"ContainerDied","Data":"3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132"}
Oct 30 04:47:40.778652 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.778498    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"
Oct 30 04:47:40.778652 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.778529    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7" event={"ID":"bd3a1d2a-bd77-4141-a4c2-ad672da165ee","Type":"ContainerDied","Data":"7ff1e0ba240ef0a5b7991f72d59773d96e3ce58432727ab86ea4f6c3c476a81a"}
Oct 30 04:47:40.778652 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.778551    5072 scope.go:122] "RemoveContainer" containerID="3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132"
Oct 30 04:47:40.790440 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.790420    5072 scope.go:122] "RemoveContainer" containerID="3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132"
Oct 30 04:47:40.790719 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:47:40.790697    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132\": container with ID starting with 3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132 not found: ID does not exist" containerID="3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132"
Oct 30 04:47:40.790788 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.790726    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132"} err="failed to get container status \"3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132\": rpc error: code = NotFound desc = could not find container \"3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132\": container with ID starting with 3582e6281f81dcdbdbf7ee5151ac6f508a96abd38be0ab27cfe0f5649d09d132 not found: ID does not exist"
Oct 30 04:47:40.836834 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.836798    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"]
Oct 30 04:47:40.848283 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:40.848262    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-fd69vl2p7"]
Oct 30 04:47:41.712056 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:41.712027    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:47:41.752201 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:47:41.752173    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:47:42.410503 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:42.410474    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3a1d2a-bd77-4141-a4c2-ad672da165ee" path="/var/lib/kubelet/pods/bd3a1d2a-bd77-4141-a4c2-ad672da165ee/volumes"
Oct 30 04:47:46.752925 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:46.752892    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:47:47.737373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.737337    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"]
Oct 30 04:47:47.738062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.738048    5072 cpu_manager.go:414] "containerMap: removing container" podUID="bd3a1d2a-bd77-4141-a4c2-ad672da165ee" containerName="payload-processing"
Oct 30 04:47:47.738161 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.738063    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3a1d2a-bd77-4141-a4c2-ad672da165ee" containerName="payload-processing"
Oct 30 04:47:47.738248 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.738237    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3a1d2a-bd77-4141-a4c2-ad672da165ee" containerName="payload-processing"
Oct 30 04:47:47.743366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.743347    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:47.763523 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.763499    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"]
Oct 30 04:47:47.770417 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.770392    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"kube-root-ca.crt\""
Oct 30 04:47:47.770622 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.770392    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-91705f\""
Oct 30 04:47:47.770735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.770711    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"openshift-service-ca.crt\""
Oct 30 04:47:47.770803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.770471    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"agc-cred-ec6aabb684fc\""
Oct 30 04:47:47.775164 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.775146    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"routing-overlay\""
Oct 30 04:47:47.775274 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.775242    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"payload-processing-external-model-e2e--a5b23dec-dockercfg-4b2z5\""
Oct 30 04:47:47.844302 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.844240    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:47.844516 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.844325    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:47.844516 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.844366    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95vn\" (UniqueName: \"kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-kube-api-access-n95vn\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:47.844516 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.844406    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:47.946333 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.946303    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:47.946544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.946339    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:47.946544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.946375    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95vn\" (UniqueName: \"kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-kube-api-access-n95vn\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:47.946544 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.946507    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:47.947135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.947109    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:47.947268 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.947219    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:47.949066 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.949046    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:47.975465 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:47.975444    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95vn\" (UniqueName: \"kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-kube-api-access-n95vn\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:48.058673 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:48.058592    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:48.365838 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:47:48.365804    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e30a5b_5ae8_4efb_8459_c0e094851980.slice/crio-5ae43966f2626096de5478e5df9099cea5ebebffe74e688e831cc5eace6b1d2b WatchSource:0}: Error finding container 5ae43966f2626096de5478e5df9099cea5ebebffe74e688e831cc5eace6b1d2b: Status 404 returned error can't find the container with id 5ae43966f2626096de5478e5df9099cea5ebebffe74e688e831cc5eace6b1d2b
Oct 30 04:47:48.366440 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:48.366397    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"]
Oct 30 04:47:48.856149 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:48.856111    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj" event={"ID":"62e30a5b-5ae8-4efb-8459-c0e094851980","Type":"ContainerStarted","Data":"6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11"}
Oct 30 04:47:48.856568 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:48.856171    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj" event={"ID":"62e30a5b-5ae8-4efb-8459-c0e094851980","Type":"ContainerStarted","Data":"5ae43966f2626096de5478e5df9099cea5ebebffe74e688e831cc5eace6b1d2b"}
Oct 30 04:47:48.856568 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:48.856240    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:47:48.902390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:48.902344    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj" podStartSLOduration=1.902330571 podStartE2EDuration="1.902330571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:47:48.899329103 +0000 UTC m=+3048.640752201" watchObservedRunningTime="2026-10-30 04:47:48.902330571 +0000 UTC m=+3048.643753667"
Oct 30 04:47:51.910978 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:51.910944    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"]
Oct 30 04:47:51.915774 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:51.915751    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:51.938620 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:51.938590    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"]
Oct 30 04:47:51.943507 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:51.943484    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"routing-overlay\""
Oct 30 04:47:51.943780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:51.943762    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"openshift-service-ca.crt\""
Oct 30 04:47:51.943913 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:51.943888    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"kube-root-ca.crt\""
Oct 30 04:47:51.948431 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:51.943991    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"e2e-cmp-legacy-20f5e7-provider-cred\""
Oct 30 04:47:51.948532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:51.943774    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ed8266\""
Oct 30 04:47:51.948532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:51.945876    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"payload-processing-external-model-e2e--9d9c1ba6-dockercfg-6q5f9\""
Oct 30 04:47:52.005204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.005175    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.005347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.005324    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.005404 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.005376    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.005497 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.005476    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwt4v\" (UniqueName: \"kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-kube-api-access-kwt4v\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.106789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.106751    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwt4v\" (UniqueName: \"kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-kube-api-access-kwt4v\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.107001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.106805    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.107001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.106931    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.107001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.106978    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.107621 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.107592    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.107725 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.107647    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.109633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.109614    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.130617 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.130593    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwt4v\" (UniqueName: \"kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-kube-api-access-kwt4v\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.230923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.230894    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.536491 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:47:52.536457    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a81b73_3cb5_4f40_b91b_fb824821c9f2.slice/crio-0ef6368f955a88a0444d6c5888b53907708013c0dbc8464c55af570bdd70a681 WatchSource:0}: Error finding container 0ef6368f955a88a0444d6c5888b53907708013c0dbc8464c55af570bdd70a681: Status 404 returned error can't find the container with id 0ef6368f955a88a0444d6c5888b53907708013c0dbc8464c55af570bdd70a681
Oct 30 04:47:52.537524 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.537500    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"]
Oct 30 04:47:52.888090 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.887989    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6" event={"ID":"26a81b73-3cb5-4f40-b91b-fb824821c9f2","Type":"ContainerStarted","Data":"0496999182e2e4cbf1c3105f0b3d99a77c234a48f69ecb8cf2f63d6ef85484cd"}
Oct 30 04:47:52.888090 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.888037    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6" event={"ID":"26a81b73-3cb5-4f40-b91b-fb824821c9f2","Type":"ContainerStarted","Data":"0ef6368f955a88a0444d6c5888b53907708013c0dbc8464c55af570bdd70a681"}
Oct 30 04:47:52.888318 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.888226    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:47:52.993760 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:52.993702    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6" podStartSLOduration=1.993688631 podStartE2EDuration="1.993688631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:47:52.990323536 +0000 UTC m=+3052.731746634" watchObservedRunningTime="2026-10-30 04:47:52.993688631 +0000 UTC m=+3052.735111728"
Oct 30 04:47:59.866954 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:47:59.866920    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:48:03.899506 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:03.899474    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:48:10.030978 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:10.030946    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"]
Oct 30 04:48:10.043492 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:10.043466    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d58b6b8b-srgh5"]
Oct 30 04:48:10.409555 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:10.409477    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="002b0525-c6d4-4305-bb59-e3e470aaaff3" path="/var/lib/kubelet/pods/002b0525-c6d4-4305-bb59-e3e470aaaff3/volumes"
Oct 30 04:48:11.754136 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:48:11.754100    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:48:20.047387 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:20.047345    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"]
Oct 30 04:48:20.056356 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:20.056322    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c180b0f-cgpfx"]
Oct 30 04:48:20.411007 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:20.410917    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf30567-3169-4681-91a2-8185771bd570" path="/var/lib/kubelet/pods/7bf30567-3169-4681-91a2-8185771bd570/volumes"
Oct 30 04:48:22.117854 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.117811    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"]
Oct 30 04:48:22.124120 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.124055    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.140395 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.140364    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"]
Oct 30 04:48:22.148641 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.148610    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-721c85\"/\"default-dockercfg-4pcfn\""
Oct 30 04:48:22.148845 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.148824    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-721c85\"/\"autores-model-721c85-kserve-self-signed-certs\""
Oct 30 04:48:22.153472 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.153443    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-721c85\"/\"kube-root-ca.crt\""
Oct 30 04:48:22.153638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.153509    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-721c85\"/\"openshift-service-ca.crt\""
Oct 30 04:48:22.284069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.284029    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-home\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.284316 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.284213    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-dshm\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.284394 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.284364    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tls-certs\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.284465 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.284406    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-model-cache\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.284591 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.284560    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6tcq\" (UniqueName: \"kubernetes.io/projected/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-kube-api-access-v6tcq\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.284780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.284711    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tmp-dir\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.386509 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.386417    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tls-certs\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.386783 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.386754    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-model-cache\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.386975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.386957    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6tcq\" (UniqueName: \"kubernetes.io/projected/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-kube-api-access-v6tcq\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.387247 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.387223    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-model-cache\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.387407 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.387234    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tmp-dir\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.387533 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.387505    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tmp-dir\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.387657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.387566    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-home\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.387657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.387643    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-dshm\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.387924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.387901    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-home\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.390901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.390879    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tls-certs\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.392002 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.391979    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-dshm\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.412133 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.412105    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6tcq\" (UniqueName: \"kubernetes.io/projected/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-kube-api-access-v6tcq\") pod \"autores-model-721c85-kserve-b5b65bb97-z4nhr\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") " pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.442468 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.442359    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:22.749714 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:48:22.749681    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc63d3bd0_d643_4880_9847_3c2ebbc5d11f.slice/crio-2447f4510f8135a23435d538a7ecd8c1788975c511580b837d28a66e61169d5d WatchSource:0}: Error finding container 2447f4510f8135a23435d538a7ecd8c1788975c511580b837d28a66e61169d5d: Status 404 returned error can't find the container with id 2447f4510f8135a23435d538a7ecd8c1788975c511580b837d28a66e61169d5d
Oct 30 04:48:22.752501 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:22.752475    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"]
Oct 30 04:48:23.110541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:23.110443    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr" event={"ID":"c63d3bd0-d643-4880-9847-3c2ebbc5d11f","Type":"ContainerStarted","Data":"7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25"}
Oct 30 04:48:23.110541 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:23.110490    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr" event={"ID":"c63d3bd0-d643-4880-9847-3c2ebbc5d11f","Type":"ContainerStarted","Data":"2447f4510f8135a23435d538a7ecd8c1788975c511580b837d28a66e61169d5d"}
Oct 30 04:48:23.155258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:23.155203    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr" podStartSLOduration=1.155189954 podStartE2EDuration="1.155189954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:48:23.153799114 +0000 UTC m=+3082.895222212" watchObservedRunningTime="2026-10-30 04:48:23.155189954 +0000 UTC m=+3082.896613051"
Oct 30 04:48:23.442731 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:23.442700    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:32.442744 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:32.442710    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:32.455795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:32.455768    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:33.197067 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:33.197039    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:48:40.869248 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:40.869163    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"]
Oct 30 04:48:40.873799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:40.873779    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:40.902463 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:40.902435    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"agc-cred-1b38359d0333\""
Oct 30 04:48:40.908426 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:40.908391    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"]
Oct 30 04:48:41.014333 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.014297    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzvs8\" (UniqueName: \"kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-kube-api-access-dzvs8\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.014517 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.014376    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.014566 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.014546    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.014606 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.014592    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.116130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.116091    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvs8\" (UniqueName: \"kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-kube-api-access-dzvs8\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.116337 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.116147    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.116337 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.116233    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.116337 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.116269    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.117034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.117002    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.117158 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.117018    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.118961 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.118938    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.140252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.140196    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzvs8\" (UniqueName: \"kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-kube-api-access-dzvs8\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.189281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.189249    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:41.519447 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:48:41.519411    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b5759d_ff2e_43c9_981f_5c04b1131ebf.slice/crio-b473e7de971fb7ef438a12520e2e7d9eab82707fea28c458b9d8dd97f80c102a WatchSource:0}: Error finding container b473e7de971fb7ef438a12520e2e7d9eab82707fea28c458b9d8dd97f80c102a: Status 404 returned error can't find the container with id b473e7de971fb7ef438a12520e2e7d9eab82707fea28c458b9d8dd97f80c102a
Oct 30 04:48:41.520945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.520917    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"]
Oct 30 04:48:41.712114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.712063    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"]
Oct 30 04:48:41.716701 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.716681    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:41.734267 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.734244    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"agc-cred-0bafc3e7a90c\""
Oct 30 04:48:41.738950 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.738922    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"]
Oct 30 04:48:41.764165 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:48:41.764131    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:48:41.826348 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.826284    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:41.826348 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.826324    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ldmk\" (UniqueName: \"kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-kube-api-access-4ldmk\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:41.826565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.826468    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:41.826687 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.826671    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:41.927986 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.927955    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:41.928463 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.928003    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ldmk\" (UniqueName: \"kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-kube-api-access-4ldmk\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:41.928463 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.928060    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:41.928463 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.928260    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:41.928639 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.928616    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:41.928841 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.928821    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:41.930880 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.930856    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:41.952234 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:41.952214    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ldmk\" (UniqueName: \"kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-kube-api-access-4ldmk\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:42.033632 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:42.033607    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:42.257894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:42.257849    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc" event={"ID":"44b5759d-ff2e-43c9-981f-5c04b1131ebf","Type":"ContainerStarted","Data":"491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd"}
Oct 30 04:48:42.258116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:42.257913    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:42.258116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:42.257932    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc" event={"ID":"44b5759d-ff2e-43c9-981f-5c04b1131ebf","Type":"ContainerStarted","Data":"b473e7de971fb7ef438a12520e2e7d9eab82707fea28c458b9d8dd97f80c102a"}
Oct 30 04:48:42.302173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:42.302117    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc" podStartSLOduration=2.302068412 podStartE2EDuration="2.302068412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:48:42.300323996 +0000 UTC m=+3102.041747094" watchObservedRunningTime="2026-10-30 04:48:42.302068412 +0000 UTC m=+3102.043491536"
Oct 30 04:48:42.343702 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:48:42.343639    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb821e187_e48f_4ce4_92fd_b0185252d142.slice/crio-558602961bdf903ec7939be6c7d04d790934b6e1b630b61d6f4534ec571934e6 WatchSource:0}: Error finding container 558602961bdf903ec7939be6c7d04d790934b6e1b630b61d6f4534ec571934e6: Status 404 returned error can't find the container with id 558602961bdf903ec7939be6c7d04d790934b6e1b630b61d6f4534ec571934e6
Oct 30 04:48:42.344486 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:42.344463    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"]
Oct 30 04:48:43.267143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:43.267066    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw" event={"ID":"b821e187-e48f-4ce4-92fd-b0185252d142","Type":"ContainerStarted","Data":"b8aae1c1a3f4f4dc634ce9a91eb04ed1f7b17e2a58ba0a36f9fec4d014d0cc2c"}
Oct 30 04:48:43.267665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:43.267176    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw" event={"ID":"b821e187-e48f-4ce4-92fd-b0185252d142","Type":"ContainerStarted","Data":"558602961bdf903ec7939be6c7d04d790934b6e1b630b61d6f4534ec571934e6"}
Oct 30 04:48:43.267665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:43.267378    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:43.324606 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:43.324553    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw" podStartSLOduration=2.3245400800000002 podStartE2EDuration="2.32454008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:48:43.322481394 +0000 UTC m=+3103.063904491" watchObservedRunningTime="2026-10-30 04:48:43.32454008 +0000 UTC m=+3103.065963177"
Oct 30 04:48:47.632054 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:47.632019    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"]
Oct 30 04:48:53.271285 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:53.271257    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:48:53.351220 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:53.351184    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"]
Oct 30 04:48:54.277815 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:54.277782    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:48:54.355004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:54.354966    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"]
Oct 30 04:48:58.352086 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.352045    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6" podUID="700b211e-c846-4d46-96eb-b2ef28e93c0e" containerName="payload-processing" containerID="cri-o://b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256" gracePeriod=25
Oct 30 04:48:58.634657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.634624    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:48:58.728380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.728348    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-routing-overlay-volume\") pod \"700b211e-c846-4d46-96eb-b2ef28e93c0e\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") "
Oct 30 04:48:58.728548 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.728413    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-kube-api-access-xgvf7\" (UniqueName: \"kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-kube-api-access-xgvf7\") pod \"700b211e-c846-4d46-96eb-b2ef28e93c0e\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") "
Oct 30 04:48:58.728548 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.728507    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-provider-credentials\") pod \"700b211e-c846-4d46-96eb-b2ef28e93c0e\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") "
Oct 30 04:48:58.728670 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.728653    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-plugins-config-volume\") pod \"700b211e-c846-4d46-96eb-b2ef28e93c0e\" (UID: \"700b211e-c846-4d46-96eb-b2ef28e93c0e\") "
Oct 30 04:48:58.728706 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.728680    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-routing-overlay-volume" pod "700b211e-c846-4d46-96eb-b2ef28e93c0e" (UID: "700b211e-c846-4d46-96eb-b2ef28e93c0e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:48:58.729003 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.728979    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-plugins-config-volume" pod "700b211e-c846-4d46-96eb-b2ef28e93c0e" (UID: "700b211e-c846-4d46-96eb-b2ef28e93c0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:48:58.729200 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.729173    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:48:58.729200 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.729188    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/700b211e-c846-4d46-96eb-b2ef28e93c0e-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:48:58.730827 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.730800    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-provider-credentials" pod "700b211e-c846-4d46-96eb-b2ef28e93c0e" (UID: "700b211e-c846-4d46-96eb-b2ef28e93c0e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:48:58.730915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.730828    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-kube-api-access-xgvf7" pod "700b211e-c846-4d46-96eb-b2ef28e93c0e" (UID: "700b211e-c846-4d46-96eb-b2ef28e93c0e"). InnerVolumeSpecName "kube-api-access-xgvf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:48:58.830836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.830801    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgvf7\" (UniqueName: \"kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-kube-api-access-xgvf7\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:48:58.830836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:58.830827    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/700b211e-c846-4d46-96eb-b2ef28e93c0e-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:48:59.355327 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.355278    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj" podUID="9ca39491-d862-43a9-a037-1f8b994c91a9" containerName="payload-processing" containerID="cri-o://1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831" gracePeriod=25
Oct 30 04:48:59.402320 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.402291    5072 generic.go:356] "Generic (PLEG): container finished" podID="700b211e-c846-4d46-96eb-b2ef28e93c0e" containerID="b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256" exitCode=0
Oct 30 04:48:59.402436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.402336    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"
Oct 30 04:48:59.402436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.402344    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6" event={"ID":"700b211e-c846-4d46-96eb-b2ef28e93c0e","Type":"ContainerDied","Data":"b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256"}
Oct 30 04:48:59.402547 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.402454    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6" event={"ID":"700b211e-c846-4d46-96eb-b2ef28e93c0e","Type":"ContainerDied","Data":"4b02c9134e4becd5f819ca5c784ddeb7f0950a5210bdf22cc69b9ab1354f8acb"}
Oct 30 04:48:59.402547 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.402475    5072 scope.go:122] "RemoveContainer" containerID="b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256"
Oct 30 04:48:59.478677 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.478654    5072 scope.go:122] "RemoveContainer" containerID="b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256"
Oct 30 04:48:59.479286 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:48:59.479258    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256\": container with ID starting with b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256 not found: ID does not exist" containerID="b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256"
Oct 30 04:48:59.479420 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.479296    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256"} err="failed to get container status \"b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256\": rpc error: code = NotFound desc = could not find container \"b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256\": container with ID starting with b4bc6da50e29e828dc8865437f176edd589167a9fc6006837ed0df5b4a21a256 not found: ID does not exist"
Oct 30 04:48:59.494135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.494104    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"]
Oct 30 04:48:59.507153 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.507124    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95nmmk6"]
Oct 30 04:48:59.645974 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.645948    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:48:59.742025 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.741989    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-routing-overlay-volume\") pod \"9ca39491-d862-43a9-a037-1f8b994c91a9\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") "
Oct 30 04:48:59.742251 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.742130    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-kube-api-access-zkrvj\" (UniqueName: \"kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-kube-api-access-zkrvj\") pod \"9ca39491-d862-43a9-a037-1f8b994c91a9\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") "
Oct 30 04:48:59.742251 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.742214    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-plugins-config-volume\") pod \"9ca39491-d862-43a9-a037-1f8b994c91a9\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") "
Oct 30 04:48:59.742368 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.742297    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-provider-credentials\") pod \"9ca39491-d862-43a9-a037-1f8b994c91a9\" (UID: \"9ca39491-d862-43a9-a037-1f8b994c91a9\") "
Oct 30 04:48:59.742561 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.742537    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-routing-overlay-volume" pod "9ca39491-d862-43a9-a037-1f8b994c91a9" (UID: "9ca39491-d862-43a9-a037-1f8b994c91a9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:48:59.742677 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.742657    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-plugins-config-volume" pod "9ca39491-d862-43a9-a037-1f8b994c91a9" (UID: "9ca39491-d862-43a9-a037-1f8b994c91a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:48:59.742737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.742725    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:48:59.744592 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.744565    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-kube-api-access-zkrvj" pod "9ca39491-d862-43a9-a037-1f8b994c91a9" (UID: "9ca39491-d862-43a9-a037-1f8b994c91a9"). InnerVolumeSpecName "kube-api-access-zkrvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:48:59.744705 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.744662    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-provider-credentials" pod "9ca39491-d862-43a9-a037-1f8b994c91a9" (UID: "9ca39491-d862-43a9-a037-1f8b994c91a9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:48:59.844134 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.844068    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkrvj\" (UniqueName: \"kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-kube-api-access-zkrvj\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:48:59.844134 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.844134    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca39491-d862-43a9-a037-1f8b994c91a9-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:48:59.844134 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:48:59.844144    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ca39491-d862-43a9-a037-1f8b994c91a9-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:00.423295 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:00.423263    5072 generic.go:356] "Generic (PLEG): container finished" podID="9ca39491-d862-43a9-a037-1f8b994c91a9" containerID="1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831" exitCode=0
Oct 30 04:49:00.423794 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:00.423410    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"
Oct 30 04:49:00.423794 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:00.423648    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700b211e-c846-4d46-96eb-b2ef28e93c0e" path="/var/lib/kubelet/pods/700b211e-c846-4d46-96eb-b2ef28e93c0e/volumes"
Oct 30 04:49:00.424323 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:00.424246    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj" event={"ID":"9ca39491-d862-43a9-a037-1f8b994c91a9","Type":"ContainerDied","Data":"1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831"}
Oct 30 04:49:00.424323 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:00.424296    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj" event={"ID":"9ca39491-d862-43a9-a037-1f8b994c91a9","Type":"ContainerDied","Data":"2b1725ec7e05e891f047f932eceab53374402284928dc62afff10b1fbfdbbc22"}
Oct 30 04:49:00.424323 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:00.424325    5072 scope.go:122] "RemoveContainer" containerID="1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831"
Oct 30 04:49:00.438315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:00.438297    5072 scope.go:122] "RemoveContainer" containerID="1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831"
Oct 30 04:49:00.438637 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:49:00.438612    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831\": container with ID starting with 1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831 not found: ID does not exist" containerID="1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831"
Oct 30 04:49:00.438737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:00.438644    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831"} err="failed to get container status \"1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831\": rpc error: code = NotFound desc = could not find container \"1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831\": container with ID starting with 1bcde33cb88642f544c3d53ca05ff71916ef95ba8c71eb7a93426d77787ac831 not found: ID does not exist"
Oct 30 04:49:00.481818 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:00.481784    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"]
Oct 30 04:49:00.492105 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:00.492060    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d8567lvj"]
Oct 30 04:49:02.409719 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.409689    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca39491-d862-43a9-a037-1f8b994c91a9" path="/var/lib/kubelet/pods/9ca39491-d862-43a9-a037-1f8b994c91a9/volumes"
Oct 30 04:49:02.655463 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.655413    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr" podUID="c63d3bd0-d643-4880-9847-3c2ebbc5d11f" containerName="main" containerID="cri-o://7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25" gracePeriod=45
Oct 30 04:49:02.938057 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.938031    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:49:02.985180 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.985140    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tls-certs\") pod \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") "
Oct 30 04:49:02.985353 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.985214    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-model-cache\") pod \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") "
Oct 30 04:49:02.985353 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.985296    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-dshm\") pod \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") "
Oct 30 04:49:02.985353 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.985324    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tmp-dir\") pod \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") "
Oct 30 04:49:02.985532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.985405    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-kube-api-access-v6tcq\" (UniqueName: \"kubernetes.io/projected/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-kube-api-access-v6tcq\") pod \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") "
Oct 30 04:49:02.985532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.985449    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-home\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-home\") pod \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\" (UID: \"c63d3bd0-d643-4880-9847-3c2ebbc5d11f\") "
Oct 30 04:49:02.986159 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.986114    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-model-cache" pod "c63d3bd0-d643-4880-9847-3c2ebbc5d11f" (UID: "c63d3bd0-d643-4880-9847-3c2ebbc5d11f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:49:02.986296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.986208    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-home" pod "c63d3bd0-d643-4880-9847-3c2ebbc5d11f" (UID: "c63d3bd0-d643-4880-9847-3c2ebbc5d11f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:49:02.986413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.986385    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tmp-dir" pod "c63d3bd0-d643-4880-9847-3c2ebbc5d11f" (UID: "c63d3bd0-d643-4880-9847-3c2ebbc5d11f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:49:02.986473 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.986436    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:02.986473 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.986454    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:02.999618 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.999589    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-dshm" pod "c63d3bd0-d643-4880-9847-3c2ebbc5d11f" (UID: "c63d3bd0-d643-4880-9847-3c2ebbc5d11f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:49:02.999618 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:02.999607    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tls-certs" pod "c63d3bd0-d643-4880-9847-3c2ebbc5d11f" (UID: "c63d3bd0-d643-4880-9847-3c2ebbc5d11f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:49:03.001514 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.001486    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-kube-api-access-v6tcq" pod "c63d3bd0-d643-4880-9847-3c2ebbc5d11f" (UID: "c63d3bd0-d643-4880-9847-3c2ebbc5d11f"). InnerVolumeSpecName "kube-api-access-v6tcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:03.087333 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.087302    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:03.087333 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.087331    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:03.087508 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.087341    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6tcq\" (UniqueName: \"kubernetes.io/projected/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-kube-api-access-v6tcq\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:03.087508 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.087352    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63d3bd0-d643-4880-9847-3c2ebbc5d11f-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:03.450897 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.450863    5072 generic.go:356] "Generic (PLEG): container finished" podID="c63d3bd0-d643-4880-9847-3c2ebbc5d11f" containerID="7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25" exitCode=0
Oct 30 04:49:03.451376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.450898    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr" event={"ID":"c63d3bd0-d643-4880-9847-3c2ebbc5d11f","Type":"ContainerDied","Data":"7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25"}
Oct 30 04:49:03.451376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.450938    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"
Oct 30 04:49:03.451376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.450957    5072 scope.go:122] "RemoveContainer" containerID="7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25"
Oct 30 04:49:03.451376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.450945    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr" event={"ID":"c63d3bd0-d643-4880-9847-3c2ebbc5d11f","Type":"ContainerDied","Data":"2447f4510f8135a23435d538a7ecd8c1788975c511580b837d28a66e61169d5d"}
Oct 30 04:49:03.461642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.461628    5072 scope.go:122] "RemoveContainer" containerID="7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25"
Oct 30 04:49:03.461892 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:49:03.461875    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25\": container with ID starting with 7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25 not found: ID does not exist" containerID="7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25"
Oct 30 04:49:03.461946 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.461900    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25"} err="failed to get container status \"7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25\": rpc error: code = NotFound desc = could not find container \"7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25\": container with ID starting with 7fa5a7369e2306b08c63470b1d104d00148bbd457c9c2b33c7709a1daa861d25 not found: ID does not exist"
Oct 30 04:49:03.505241 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.505218    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"]
Oct 30 04:49:03.517583 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:03.517556    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-721c85/autores-model-721c85-kserve-b5b65bb97-z4nhr"]
Oct 30 04:49:04.410218 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:04.410181    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63d3bd0-d643-4880-9847-3c2ebbc5d11f" path="/var/lib/kubelet/pods/c63d3bd0-d643-4880-9847-3c2ebbc5d11f/volumes"
Oct 30 04:49:05.368405 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.368369    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"]
Oct 30 04:49:05.369285 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.369122    5072 cpu_manager.go:414] "containerMap: removing container" podUID="700b211e-c846-4d46-96eb-b2ef28e93c0e" containerName="payload-processing"
Oct 30 04:49:05.369285 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.369139    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700b211e-c846-4d46-96eb-b2ef28e93c0e" containerName="payload-processing"
Oct 30 04:49:05.369285 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.369152    5072 cpu_manager.go:414] "containerMap: removing container" podUID="9ca39491-d862-43a9-a037-1f8b994c91a9" containerName="payload-processing"
Oct 30 04:49:05.369285 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.369157    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca39491-d862-43a9-a037-1f8b994c91a9" containerName="payload-processing"
Oct 30 04:49:05.369285 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.369166    5072 cpu_manager.go:414] "containerMap: removing container" podUID="c63d3bd0-d643-4880-9847-3c2ebbc5d11f" containerName="main"
Oct 30 04:49:05.369285 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.369171    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63d3bd0-d643-4880-9847-3c2ebbc5d11f" containerName="main"
Oct 30 04:49:05.369527 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.369303    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="700b211e-c846-4d46-96eb-b2ef28e93c0e" containerName="payload-processing"
Oct 30 04:49:05.369527 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.369312    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca39491-d862-43a9-a037-1f8b994c91a9" containerName="payload-processing"
Oct 30 04:49:05.369527 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.369318    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63d3bd0-d643-4880-9847-3c2ebbc5d11f" containerName="main"
Oct 30 04:49:05.374366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.374348    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.400847 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.400814    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"]
Oct 30 04:49:05.406798 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.406755    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"agc-cred-a374b527451b\""
Oct 30 04:49:05.406923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.406858    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6378cc\""
Oct 30 04:49:05.406923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.406862    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"openshift-service-ca.crt\""
Oct 30 04:49:05.407197 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.407172    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"routing-overlay\""
Oct 30 04:49:05.407293 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.407198    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"kube-root-ca.crt\""
Oct 30 04:49:05.411113 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.411088    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"payload-processing-external-model-e2e--18641857-dockercfg-kfsbt\""
Oct 30 04:49:05.420270 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.420248    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g99dk\" (UniqueName: \"kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-kube-api-access-g99dk\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.420444 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.420410    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.420543 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.420521    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.420647 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.420632    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.521680 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.521643    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.521852 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.521767    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g99dk\" (UniqueName: \"kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-kube-api-access-g99dk\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.521910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.521891    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.521953 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.521937    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.522616 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.522579    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.522747 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.522731    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.524656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.524632    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.545981 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.545946    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g99dk\" (UniqueName: \"kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-kube-api-access-g99dk\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:05.688896 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:05.688870    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:06.001191 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.001138    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"]
Oct 30 04:49:06.002258 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:06.002220    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb92f7ed_2b60_4ca7_93e4_072c0cb770b5.slice/crio-b7435588f3e231f4809569372a40a4de3af0a246dd9068a903d745f232fe9515 WatchSource:0}: Error finding container b7435588f3e231f4809569372a40a4de3af0a246dd9068a903d745f232fe9515: Status 404 returned error can't find the container with id b7435588f3e231f4809569372a40a4de3af0a246dd9068a903d745f232fe9515
Oct 30 04:49:06.298964 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.298886    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"]
Oct 30 04:49:06.303574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.303554    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.324026 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.324003    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"]
Oct 30 04:49:06.331126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.331097    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"payload-processing-external-model-plugins-e2e-extmod-a-05bee8\""
Oct 30 04:49:06.331256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.331246    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"openshift-service-ca.crt\""
Oct 30 04:49:06.340349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.340314    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"kube-root-ca.crt\""
Oct 30 04:49:06.340349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.340326    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"routing-overlay\""
Oct 30 04:49:06.340349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.340333    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"agc-cred-5426296d5e69\""
Oct 30 04:49:06.340529 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.340375    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"payload-processing-external-model-e2e--545d7aee-dockercfg-gcdhs\""
Oct 30 04:49:06.434892 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.434858    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.434892 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.434892    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.435363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.435057    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.435363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.435112    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9wmr\" (UniqueName: \"kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-kube-api-access-q9wmr\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.493456 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.493420    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx" event={"ID":"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5","Type":"ContainerStarted","Data":"42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46"}
Oct 30 04:49:06.493616 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.493479    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx" event={"ID":"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5","Type":"ContainerStarted","Data":"b7435588f3e231f4809569372a40a4de3af0a246dd9068a903d745f232fe9515"}
Oct 30 04:49:06.493616 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.493523    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:06.536258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.536233    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.536432 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.536266    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.536432 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.536411    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.536565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.536443    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9wmr\" (UniqueName: \"kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-kube-api-access-q9wmr\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.537131 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.537105    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.537241 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.537136    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.537241 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.537147    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx" podStartSLOduration=1.5371339179999999 podStartE2EDuration="1.537133918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:06.535912834 +0000 UTC m=+3126.277335931" watchObservedRunningTime="2026-10-30 04:49:06.537133918 +0000 UTC m=+3126.278557015"
Oct 30 04:49:06.539577 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.539556    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.562238 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.562164    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9wmr\" (UniqueName: \"kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-kube-api-access-q9wmr\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.619058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.619024    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:06.944503 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:06.944473    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ed81c9_f26d_4045_8180_835b41ddd212.slice/crio-b0be4f5cf70494645b2d5c32127bac4488a06d07c1e003eb1f8095267808c438 WatchSource:0}: Error finding container b0be4f5cf70494645b2d5c32127bac4488a06d07c1e003eb1f8095267808c438: Status 404 returned error can't find the container with id b0be4f5cf70494645b2d5c32127bac4488a06d07c1e003eb1f8095267808c438
Oct 30 04:49:06.946491 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.946469    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"]
Oct 30 04:49:06.978443 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.978417    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"]
Oct 30 04:49:06.984467 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:06.984449    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.009146 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.009115    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"agc-cred-e68b135ffabe\""
Oct 30 04:49:07.014536 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.014512    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"]
Oct 30 04:49:07.044156 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.044126    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-848d98lsh\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.044289 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.044195    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-848d98lsh\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.044351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.044317    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-848d98lsh\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.044471 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.044456    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sbls\" (UniqueName: \"kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-kube-api-access-9sbls\") pod \"payload-processing-external-model-e2e-extmod-b-503804-848d98lsh\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.145699 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.145667    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sbls\" (UniqueName: \"kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-kube-api-access-9sbls\") pod \"payload-processing-external-model-e2e-extmod-b-503804-848d98lsh\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.145830 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.145745    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-848d98lsh\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.145830 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.145780    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-848d98lsh\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.145927 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.145841    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-848d98lsh\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.146683 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.146601    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-848d98lsh\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.146683 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.146639    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-848d98lsh\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.148566 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.148547    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-848d98lsh\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.175566 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.175544    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sbls\" (UniqueName: \"kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-kube-api-access-9sbls\") pod \"payload-processing-external-model-e2e-extmod-b-503804-848d98lsh\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.302377 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.302301    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:07.501841 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.501806    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w" event={"ID":"71ed81c9-f26d-4045-8180-835b41ddd212","Type":"ContainerStarted","Data":"5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d"}
Oct 30 04:49:07.502297 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.501868    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w" event={"ID":"71ed81c9-f26d-4045-8180-835b41ddd212","Type":"ContainerStarted","Data":"b0be4f5cf70494645b2d5c32127bac4488a06d07c1e003eb1f8095267808c438"}
Oct 30 04:49:07.502297 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.502104    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:07.549056 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.549001    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w" podStartSLOduration=1.5489873520000002 podStartE2EDuration="1.548987352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:07.544992805 +0000 UTC m=+3127.286415902" watchObservedRunningTime="2026-10-30 04:49:07.548987352 +0000 UTC m=+3127.290410449"
Oct 30 04:49:07.610699 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:07.610032    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c31590c_7aee_47d9_9e1e_1dd95994401a.slice/crio-6c764407727454d5dd7a10c050e9615591f5581c750c95ed7f90eb01718478b5 WatchSource:0}: Error finding container 6c764407727454d5dd7a10c050e9615591f5581c750c95ed7f90eb01718478b5: Status 404 returned error can't find the container with id 6c764407727454d5dd7a10c050e9615591f5581c750c95ed7f90eb01718478b5
Oct 30 04:49:07.611415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:07.611382    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"]
Oct 30 04:49:08.512012 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:08.511970    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh" event={"ID":"1c31590c-7aee-47d9-9e1e-1dd95994401a","Type":"ContainerStarted","Data":"2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381"}
Oct 30 04:49:08.512506 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:08.512037    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh" event={"ID":"1c31590c-7aee-47d9-9e1e-1dd95994401a","Type":"ContainerStarted","Data":"6c764407727454d5dd7a10c050e9615591f5581c750c95ed7f90eb01718478b5"}
Oct 30 04:49:08.512506 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:08.512065    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:08.558096 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:08.558036    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh" podStartSLOduration=2.558023071 podStartE2EDuration="2.558023071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:08.555458149 +0000 UTC m=+3128.296881268" watchObservedRunningTime="2026-10-30 04:49:08.558023071 +0000 UTC m=+3128.299446172"
Oct 30 04:49:10.733500 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.733466    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"]
Oct 30 04:49:10.738529 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.738508    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:10.759417 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.759388    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"]
Oct 30 04:49:10.766583 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.766557    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:49:10.766705 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.766625    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:49:10.893596 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.893558    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53ecfec9-kqf5b\" (UID: \"028a8ab3-ae5c-49b8-a011-4cabe867d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:10.893771 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.893703    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53ecfec9-kqf5b\" (UID: \"028a8ab3-ae5c-49b8-a011-4cabe867d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:10.893860 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.893792    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp484\" (UniqueName: \"kubernetes.io/projected/028a8ab3-ae5c-49b8-a011-4cabe867d158-kube-api-access-lp484\") pod \"maas-api-revoke-sub-53ecfec9-kqf5b\" (UID: \"028a8ab3-ae5c-49b8-a011-4cabe867d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:10.995419 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.995356    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53ecfec9-kqf5b\" (UID: \"028a8ab3-ae5c-49b8-a011-4cabe867d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:10.995538 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.995437    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53ecfec9-kqf5b\" (UID: \"028a8ab3-ae5c-49b8-a011-4cabe867d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:10.995538 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.995464    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp484\" (UniqueName: \"kubernetes.io/projected/028a8ab3-ae5c-49b8-a011-4cabe867d158-kube-api-access-lp484\") pod \"maas-api-revoke-sub-53ecfec9-kqf5b\" (UID: \"028a8ab3-ae5c-49b8-a011-4cabe867d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:10.996054 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.996022    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53ecfec9-kqf5b\" (UID: \"028a8ab3-ae5c-49b8-a011-4cabe867d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:10.998057 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:10.998030    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53ecfec9-kqf5b\" (UID: \"028a8ab3-ae5c-49b8-a011-4cabe867d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:11.019557 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:11.019536    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp484\" (UniqueName: \"kubernetes.io/projected/028a8ab3-ae5c-49b8-a011-4cabe867d158-kube-api-access-lp484\") pod \"maas-api-revoke-sub-53ecfec9-kqf5b\" (UID: \"028a8ab3-ae5c-49b8-a011-4cabe867d158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:11.052521 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:11.052493    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:11.365267 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:11.365232    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028a8ab3_ae5c_49b8_a011_4cabe867d158.slice/crio-ee9ac9a8a7c37f648bce0cec45c97bcc8a3d5ff0698ebe0fa4a8683831c297df WatchSource:0}: Error finding container ee9ac9a8a7c37f648bce0cec45c97bcc8a3d5ff0698ebe0fa4a8683831c297df: Status 404 returned error can't find the container with id ee9ac9a8a7c37f648bce0cec45c97bcc8a3d5ff0698ebe0fa4a8683831c297df
Oct 30 04:49:11.365446 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:11.365422    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"]
Oct 30 04:49:11.544103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:11.544035    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b" event={"ID":"028a8ab3-ae5c-49b8-a011-4cabe867d158","Type":"ContainerStarted","Data":"ee9ac9a8a7c37f648bce0cec45c97bcc8a3d5ff0698ebe0fa4a8683831c297df"}
Oct 30 04:49:11.639006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:11.638935    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"]
Oct 30 04:49:11.752940 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:49:11.752900    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:49:12.502062 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:12.502028    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"]
Oct 30 04:49:12.505309 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:12.505287    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:12.555340 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:12.555299    5072 generic.go:356] "Generic (PLEG): container finished" podID="028a8ab3-ae5c-49b8-a011-4cabe867d158" containerID="70dfedc29bfe8ad6d5343d8da31d9340ab9a3fa98321f8223270be89d2c8d064" exitCode=0
Oct 30 04:49:12.555519 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:12.555383    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b" event={"ID":"028a8ab3-ae5c-49b8-a011-4cabe867d158","Type":"ContainerDied","Data":"70dfedc29bfe8ad6d5343d8da31d9340ab9a3fa98321f8223270be89d2c8d064"}
Oct 30 04:49:13.482199 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.482163    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"]
Oct 30 04:49:13.487001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.486980    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:13.505192 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.505170    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"]
Oct 30 04:49:13.634238 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.634176    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvk2s\" (UniqueName: \"kubernetes.io/projected/ae5e1716-636c-4077-9b7f-ae125f61ad14-kube-api-access-rvk2s\") pod \"maas-api-revoke-sub-0af8e4a9-lm88p\" (UID: \"ae5e1716-636c-4077-9b7f-ae125f61ad14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:13.634394 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.634246    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0af8e4a9-lm88p\" (UID: \"ae5e1716-636c-4077-9b7f-ae125f61ad14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:13.634394 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.634315    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0af8e4a9-lm88p\" (UID: \"ae5e1716-636c-4077-9b7f-ae125f61ad14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:13.717065 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.717040    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:13.736338 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.736274    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvk2s\" (UniqueName: \"kubernetes.io/projected/ae5e1716-636c-4077-9b7f-ae125f61ad14-kube-api-access-rvk2s\") pod \"maas-api-revoke-sub-0af8e4a9-lm88p\" (UID: \"ae5e1716-636c-4077-9b7f-ae125f61ad14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:13.736338 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.736317    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0af8e4a9-lm88p\" (UID: \"ae5e1716-636c-4077-9b7f-ae125f61ad14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:13.736500 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.736364    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0af8e4a9-lm88p\" (UID: \"ae5e1716-636c-4077-9b7f-ae125f61ad14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:13.737008 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.736987    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0af8e4a9-lm88p\" (UID: \"ae5e1716-636c-4077-9b7f-ae125f61ad14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:13.738905 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.738881    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0af8e4a9-lm88p\" (UID: \"ae5e1716-636c-4077-9b7f-ae125f61ad14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:13.763456 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.763430    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvk2s\" (UniqueName: \"kubernetes.io/projected/ae5e1716-636c-4077-9b7f-ae125f61ad14-kube-api-access-rvk2s\") pod \"maas-api-revoke-sub-0af8e4a9-lm88p\" (UID: \"ae5e1716-636c-4077-9b7f-ae125f61ad14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:13.801225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.801195    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:13.838201 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.837062    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028a8ab3-ae5c-49b8-a011-4cabe867d158-kube-api-access-lp484\" (UniqueName: \"kubernetes.io/projected/028a8ab3-ae5c-49b8-a011-4cabe867d158-kube-api-access-lp484\") pod \"028a8ab3-ae5c-49b8-a011-4cabe867d158\" (UID: \"028a8ab3-ae5c-49b8-a011-4cabe867d158\") "
Oct 30 04:49:13.838201 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.837223    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-xks-ca\") pod \"028a8ab3-ae5c-49b8-a011-4cabe867d158\" (UID: \"028a8ab3-ae5c-49b8-a011-4cabe867d158\") "
Oct 30 04:49:13.838201 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.837403    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-openshift-ca\") pod \"028a8ab3-ae5c-49b8-a011-4cabe867d158\" (UID: \"028a8ab3-ae5c-49b8-a011-4cabe867d158\") "
Oct 30 04:49:13.838201 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.837883    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-openshift-ca" pod "028a8ab3-ae5c-49b8-a011-4cabe867d158" (UID: "028a8ab3-ae5c-49b8-a011-4cabe867d158"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:13.839908 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.839867    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028a8ab3-ae5c-49b8-a011-4cabe867d158-kube-api-access-lp484" pod "028a8ab3-ae5c-49b8-a011-4cabe867d158" (UID: "028a8ab3-ae5c-49b8-a011-4cabe867d158"). InnerVolumeSpecName "kube-api-access-lp484". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:13.840317 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.840298    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-xks-ca" pod "028a8ab3-ae5c-49b8-a011-4cabe867d158" (UID: "028a8ab3-ae5c-49b8-a011-4cabe867d158"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:49:13.939225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.939191    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:13.939225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.939221    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp484\" (UniqueName: \"kubernetes.io/projected/028a8ab3-ae5c-49b8-a011-4cabe867d158-kube-api-access-lp484\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:13.939476 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:13.939236    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028a8ab3-ae5c-49b8-a011-4cabe867d158-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:14.101775 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.101702    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"]
Oct 30 04:49:14.102511 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.102496    5072 cpu_manager.go:414] "containerMap: removing container" podUID="028a8ab3-ae5c-49b8-a011-4cabe867d158" containerName="revoke-keys"
Oct 30 04:49:14.102564 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.102512    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028a8ab3-ae5c-49b8-a011-4cabe867d158" containerName="revoke-keys"
Oct 30 04:49:14.102663 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.102654    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="028a8ab3-ae5c-49b8-a011-4cabe867d158" containerName="revoke-keys"
Oct 30 04:49:14.107855 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.107833    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:14.125286 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:14.125251    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5e1716_636c_4077_9b7f_ae125f61ad14.slice/crio-4a16d9a9c893b058d6c11d30889ace3ed3a8159fcc6c439e5ff53e468eb2e040 WatchSource:0}: Error finding container 4a16d9a9c893b058d6c11d30889ace3ed3a8159fcc6c439e5ff53e468eb2e040: Status 404 returned error can't find the container with id 4a16d9a9c893b058d6c11d30889ace3ed3a8159fcc6c439e5ff53e468eb2e040
Oct 30 04:49:14.126064 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.126020    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"]
Oct 30 04:49:14.135226 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.135189    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"]
Oct 30 04:49:14.242603 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.242569    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68fpf\" (UniqueName: \"kubernetes.io/projected/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-kube-api-access-68fpf\") pod \"maas-api-revoke-sub-307e3ddc-vwctb\" (UID: \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:14.242778 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.242675    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-307e3ddc-vwctb\" (UID: \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:14.242834 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.242822    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-307e3ddc-vwctb\" (UID: \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:14.344563 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.344535    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68fpf\" (UniqueName: \"kubernetes.io/projected/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-kube-api-access-68fpf\") pod \"maas-api-revoke-sub-307e3ddc-vwctb\" (UID: \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:14.344737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.344584    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-307e3ddc-vwctb\" (UID: \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:14.344737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.344650    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-307e3ddc-vwctb\" (UID: \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:14.345337 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.345313    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-307e3ddc-vwctb\" (UID: \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:14.347186 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.347168    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-307e3ddc-vwctb\" (UID: \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:14.380990 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.380964    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68fpf\" (UniqueName: \"kubernetes.io/projected/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-kube-api-access-68fpf\") pod \"maas-api-revoke-sub-307e3ddc-vwctb\" (UID: \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:14.424870 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.424846    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:14.575189 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.574163    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"
Oct 30 04:49:14.575189 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.574217    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b" event={"ID":"028a8ab3-ae5c-49b8-a011-4cabe867d158","Type":"ContainerDied","Data":"ee9ac9a8a7c37f648bce0cec45c97bcc8a3d5ff0698ebe0fa4a8683831c297df"}
Oct 30 04:49:14.575189 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.574277    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee9ac9a8a7c37f648bce0cec45c97bcc8a3d5ff0698ebe0fa4a8683831c297df"
Oct 30 04:49:14.577466 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.577416    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p" event={"ID":"ae5e1716-636c-4077-9b7f-ae125f61ad14","Type":"ContainerStarted","Data":"4a16d9a9c893b058d6c11d30889ace3ed3a8159fcc6c439e5ff53e468eb2e040"}
Oct 30 04:49:14.808509 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:14.808473    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2a3585_7fd4_4142_afcd_1c8976cff9b4.slice/crio-e20c5ac7c877f987be83b84403c1973e9f7df6fd1c2027225ae1355baa5e249f WatchSource:0}: Error finding container e20c5ac7c877f987be83b84403c1973e9f7df6fd1c2027225ae1355baa5e249f: Status 404 returned error can't find the container with id e20c5ac7c877f987be83b84403c1973e9f7df6fd1c2027225ae1355baa5e249f
Oct 30 04:49:14.808921 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:14.808896    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"]
Oct 30 04:49:15.586612 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:15.586575    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb" event={"ID":"fb2a3585-7fd4-4142-afcd-1c8976cff9b4","Type":"ContainerStarted","Data":"86e87ff61c02c8e5d1b3e572b947172edd187379ca62400712d4aa489d090f90"}
Oct 30 04:49:15.587069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:15.586639    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb" event={"ID":"fb2a3585-7fd4-4142-afcd-1c8976cff9b4","Type":"ContainerStarted","Data":"e20c5ac7c877f987be83b84403c1973e9f7df6fd1c2027225ae1355baa5e249f"}
Oct 30 04:49:15.588491 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:15.588465    5072 generic.go:356] "Generic (PLEG): container finished" podID="ae5e1716-636c-4077-9b7f-ae125f61ad14" containerID="07efd1c81ee8bb64bdce22babf62ad630eaf81d5ead4ebb3759ccaa593705708" exitCode=0
Oct 30 04:49:15.588608 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:15.588575    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p" event={"ID":"ae5e1716-636c-4077-9b7f-ae125f61ad14","Type":"ContainerDied","Data":"07efd1c81ee8bb64bdce22babf62ad630eaf81d5ead4ebb3759ccaa593705708"}
Oct 30 04:49:15.632036 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:15.631988    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb" podStartSLOduration=1.631974086 podStartE2EDuration="1.631974086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:15.627735907 +0000 UTC m=+3135.369159007" watchObservedRunningTime="2026-10-30 04:49:15.631974086 +0000 UTC m=+3135.373397183"
Oct 30 04:49:16.596968 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.596935    5072 generic.go:356] "Generic (PLEG): container finished" podID="fb2a3585-7fd4-4142-afcd-1c8976cff9b4" containerID="86e87ff61c02c8e5d1b3e572b947172edd187379ca62400712d4aa489d090f90" exitCode=0
Oct 30 04:49:16.597490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.597023    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb" event={"ID":"fb2a3585-7fd4-4142-afcd-1c8976cff9b4","Type":"ContainerDied","Data":"86e87ff61c02c8e5d1b3e572b947172edd187379ca62400712d4aa489d090f90"}
Oct 30 04:49:16.639517 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.639471    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w" podUID="71ed81c9-f26d-4045-8180-835b41ddd212" containerName="payload-processing" containerID="cri-o://5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d" gracePeriod=25
Oct 30 04:49:16.773631 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.773601    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:16.874156 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.874059    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-xks-ca\") pod \"ae5e1716-636c-4077-9b7f-ae125f61ad14\" (UID: \"ae5e1716-636c-4077-9b7f-ae125f61ad14\") "
Oct 30 04:49:16.874332 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.874202    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-openshift-ca\") pod \"ae5e1716-636c-4077-9b7f-ae125f61ad14\" (UID: \"ae5e1716-636c-4077-9b7f-ae125f61ad14\") "
Oct 30 04:49:16.874524 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.874499    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5e1716-636c-4077-9b7f-ae125f61ad14-kube-api-access-rvk2s\" (UniqueName: \"kubernetes.io/projected/ae5e1716-636c-4077-9b7f-ae125f61ad14-kube-api-access-rvk2s\") pod \"ae5e1716-636c-4077-9b7f-ae125f61ad14\" (UID: \"ae5e1716-636c-4077-9b7f-ae125f61ad14\") "
Oct 30 04:49:16.874715 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.874589    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-openshift-ca" pod "ae5e1716-636c-4077-9b7f-ae125f61ad14" (UID: "ae5e1716-636c-4077-9b7f-ae125f61ad14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:16.875412 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.875385    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:16.876616 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.876556    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-xks-ca" pod "ae5e1716-636c-4077-9b7f-ae125f61ad14" (UID: "ae5e1716-636c-4077-9b7f-ae125f61ad14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:49:16.877054 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.877035    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5e1716-636c-4077-9b7f-ae125f61ad14-kube-api-access-rvk2s" pod "ae5e1716-636c-4077-9b7f-ae125f61ad14" (UID: "ae5e1716-636c-4077-9b7f-ae125f61ad14"). InnerVolumeSpecName "kube-api-access-rvk2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:16.914502 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.914476    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:16.976429 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.976393    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae5e1716-636c-4077-9b7f-ae125f61ad14-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:16.976429 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:16.976427    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvk2s\" (UniqueName: \"kubernetes.io/projected/ae5e1716-636c-4077-9b7f-ae125f61ad14-kube-api-access-rvk2s\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:17.077531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.077496    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-routing-overlay-volume\") pod \"71ed81c9-f26d-4045-8180-835b41ddd212\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") "
Oct 30 04:49:17.077703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.077551    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-kube-api-access-q9wmr\" (UniqueName: \"kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-kube-api-access-q9wmr\") pod \"71ed81c9-f26d-4045-8180-835b41ddd212\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") "
Oct 30 04:49:17.077703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.077578    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-plugins-config-volume\") pod \"71ed81c9-f26d-4045-8180-835b41ddd212\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") "
Oct 30 04:49:17.077703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.077682    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-provider-credentials\" (UniqueName: \"kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-provider-credentials\") pod \"71ed81c9-f26d-4045-8180-835b41ddd212\" (UID: \"71ed81c9-f26d-4045-8180-835b41ddd212\") "
Oct 30 04:49:17.077868 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.077831    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-routing-overlay-volume" pod "71ed81c9-f26d-4045-8180-835b41ddd212" (UID: "71ed81c9-f26d-4045-8180-835b41ddd212"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:17.078136 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.078093    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-plugins-config-volume" pod "71ed81c9-f26d-4045-8180-835b41ddd212" (UID: "71ed81c9-f26d-4045-8180-835b41ddd212"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:17.078231 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.078222    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:17.078291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.078237    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ed81c9-f26d-4045-8180-835b41ddd212-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:17.079961 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.079932    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-provider-credentials" pod "71ed81c9-f26d-4045-8180-835b41ddd212" (UID: "71ed81c9-f26d-4045-8180-835b41ddd212"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:17.080071 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.079978    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-kube-api-access-q9wmr" pod "71ed81c9-f26d-4045-8180-835b41ddd212" (UID: "71ed81c9-f26d-4045-8180-835b41ddd212"). InnerVolumeSpecName "kube-api-access-q9wmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:17.179121 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.179054    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9wmr\" (UniqueName: \"kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-kube-api-access-q9wmr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:17.179121 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.179114    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71ed81c9-f26d-4045-8180-835b41ddd212-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:17.503188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.503125    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx" podUID="bb92f7ed-2b60-4ca7-93e4-072c0cb770b5" containerName="payload-processing" containerID="cri-o://42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46" gracePeriod=25
Oct 30 04:49:17.607100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.607033    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p" event={"ID":"ae5e1716-636c-4077-9b7f-ae125f61ad14","Type":"ContainerDied","Data":"4a16d9a9c893b058d6c11d30889ace3ed3a8159fcc6c439e5ff53e468eb2e040"}
Oct 30 04:49:17.607579 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.607117    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a16d9a9c893b058d6c11d30889ace3ed3a8159fcc6c439e5ff53e468eb2e040"
Oct 30 04:49:17.607579 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.607192    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"
Oct 30 04:49:17.609302 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.609267    5072 generic.go:356] "Generic (PLEG): container finished" podID="71ed81c9-f26d-4045-8180-835b41ddd212" containerID="5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d" exitCode=0
Oct 30 04:49:17.609449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.609377    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"
Oct 30 04:49:17.609449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.609423    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w" event={"ID":"71ed81c9-f26d-4045-8180-835b41ddd212","Type":"ContainerDied","Data":"5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d"}
Oct 30 04:49:17.609574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.609470    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w" event={"ID":"71ed81c9-f26d-4045-8180-835b41ddd212","Type":"ContainerDied","Data":"b0be4f5cf70494645b2d5c32127bac4488a06d07c1e003eb1f8095267808c438"}
Oct 30 04:49:17.609574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.609498    5072 scope.go:122] "RemoveContainer" containerID="5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d"
Oct 30 04:49:17.627738 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.627633    5072 scope.go:122] "RemoveContainer" containerID="5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d"
Oct 30 04:49:17.630928 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:49:17.630895    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d\": container with ID starting with 5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d not found: ID does not exist" containerID="5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d"
Oct 30 04:49:17.631067 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.630931    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d"} err="failed to get container status \"5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d\": rpc error: code = NotFound desc = could not find container \"5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d\": container with ID starting with 5ed54ae547956ef54a03c5f256bbe503582dd9bf32dffc484f4d690053d00f6d not found: ID does not exist"
Oct 30 04:49:17.683102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.682892    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"]
Oct 30 04:49:17.706127 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.705457    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-5c54ztv8w"]
Oct 30 04:49:17.780152 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.780122    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:17.808156 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.808129    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:17.889407 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.889302    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-openshift-ca\") pod \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\" (UID: \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\") "
Oct 30 04:49:17.889407 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.889378    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-kube-api-access-68fpf\" (UniqueName: \"kubernetes.io/projected/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-kube-api-access-68fpf\") pod \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\" (UID: \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\") "
Oct 30 04:49:17.889810 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.889658    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-xks-ca\") pod \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\" (UID: \"fb2a3585-7fd4-4142-afcd-1c8976cff9b4\") "
Oct 30 04:49:17.891049 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.891017    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-openshift-ca" pod "fb2a3585-7fd4-4142-afcd-1c8976cff9b4" (UID: "fb2a3585-7fd4-4142-afcd-1c8976cff9b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:17.893216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.893191    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-xks-ca" pod "fb2a3585-7fd4-4142-afcd-1c8976cff9b4" (UID: "fb2a3585-7fd4-4142-afcd-1c8976cff9b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:49:17.893625 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.893605    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-kube-api-access-68fpf" pod "fb2a3585-7fd4-4142-afcd-1c8976cff9b4" (UID: "fb2a3585-7fd4-4142-afcd-1c8976cff9b4"). InnerVolumeSpecName "kube-api-access-68fpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:17.990809 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.990771    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-routing-overlay-volume\") pod \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") "
Oct 30 04:49:17.990993 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.990869    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-kube-api-access-g99dk\" (UniqueName: \"kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-kube-api-access-g99dk\") pod \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") "
Oct 30 04:49:17.990993 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.990963    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-provider-credentials\") pod \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") "
Oct 30 04:49:17.990993 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.990987    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-plugins-config-volume\") pod \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\" (UID: \"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5\") "
Oct 30 04:49:17.991227 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.991113    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-routing-overlay-volume" pod "bb92f7ed-2b60-4ca7-93e4-072c0cb770b5" (UID: "bb92f7ed-2b60-4ca7-93e4-072c0cb770b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:17.991481 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.991452    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-plugins-config-volume" pod "bb92f7ed-2b60-4ca7-93e4-072c0cb770b5" (UID: "bb92f7ed-2b60-4ca7-93e4-072c0cb770b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:17.991792 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.991547    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:17.991792 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.991562    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:17.991792 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.991571    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:17.991792 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.991581    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68fpf\" (UniqueName: \"kubernetes.io/projected/fb2a3585-7fd4-4142-afcd-1c8976cff9b4-kube-api-access-68fpf\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:17.991792 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.991589    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:17.993279 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.993256    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-provider-credentials" pod "bb92f7ed-2b60-4ca7-93e4-072c0cb770b5" (UID: "bb92f7ed-2b60-4ca7-93e4-072c0cb770b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:17.993372 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:17.993344    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-kube-api-access-g99dk" pod "bb92f7ed-2b60-4ca7-93e4-072c0cb770b5" (UID: "bb92f7ed-2b60-4ca7-93e4-072c0cb770b5"). InnerVolumeSpecName "kube-api-access-g99dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:18.092275 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.092213    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g99dk\" (UniqueName: \"kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-kube-api-access-g99dk\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:18.092275 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.092236    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:18.416324 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.416240    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ed81c9-f26d-4045-8180-835b41ddd212" path="/var/lib/kubelet/pods/71ed81c9-f26d-4045-8180-835b41ddd212/volumes"
Oct 30 04:49:18.618716 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.618685    5072 generic.go:356] "Generic (PLEG): container finished" podID="bb92f7ed-2b60-4ca7-93e4-072c0cb770b5" containerID="42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46" exitCode=0
Oct 30 04:49:18.619205 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.618744    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"
Oct 30 04:49:18.619205 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.618767    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx" event={"ID":"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5","Type":"ContainerDied","Data":"42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46"}
Oct 30 04:49:18.619205 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.618811    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx" event={"ID":"bb92f7ed-2b60-4ca7-93e4-072c0cb770b5","Type":"ContainerDied","Data":"b7435588f3e231f4809569372a40a4de3af0a246dd9068a903d745f232fe9515"}
Oct 30 04:49:18.619205 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.618838    5072 scope.go:122] "RemoveContainer" containerID="42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46"
Oct 30 04:49:18.620915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.620884    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb" event={"ID":"fb2a3585-7fd4-4142-afcd-1c8976cff9b4","Type":"ContainerDied","Data":"e20c5ac7c877f987be83b84403c1973e9f7df6fd1c2027225ae1355baa5e249f"}
Oct 30 04:49:18.621043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.620936    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e20c5ac7c877f987be83b84403c1973e9f7df6fd1c2027225ae1355baa5e249f"
Oct 30 04:49:18.621043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.620994    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"
Oct 30 04:49:18.629932 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.629916    5072 scope.go:122] "RemoveContainer" containerID="42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46"
Oct 30 04:49:18.630282 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:49:18.630262    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46\": container with ID starting with 42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46 not found: ID does not exist" containerID="42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46"
Oct 30 04:49:18.630343 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.630288    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46"} err="failed to get container status \"42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46\": rpc error: code = NotFound desc = could not find container \"42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46\": container with ID starting with 42b13a0b1c2d1eed4839707951c925c7d552dc853178dc80b40b39da6d1f4e46 not found: ID does not exist"
Oct 30 04:49:18.694650 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.694621    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"]
Oct 30 04:49:18.699839 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:18.699820    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7b6f2lpvx"]
Oct 30 04:49:19.523879 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:19.523848    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:49:19.603608 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:19.603571    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"]
Oct 30 04:49:20.410276 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:20.410245    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb92f7ed-2b60-4ca7-93e4-072c0cb770b5" path="/var/lib/kubelet/pods/bb92f7ed-2b60-4ca7-93e4-072c0cb770b5/volumes"
Oct 30 04:49:22.145393 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:22.145356    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"]
Oct 30 04:49:24.032163 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.032125    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"]
Oct 30 04:49:24.044442 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.044404    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4219c66e-l7n6z"]
Oct 30 04:49:24.047756 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.047732    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"]
Oct 30 04:49:24.050591 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.050569    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629706a9-vmj4n"]
Oct 30 04:49:24.409810 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.409730    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab420b2-4283-4d6f-9271-df62eed08062" path="/var/lib/kubelet/pods/9ab420b2-4283-4d6f-9271-df62eed08062/volumes"
Oct 30 04:49:24.410282 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.410266    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72c435c-feac-4ef5-9dce-c6f23a90e7ae" path="/var/lib/kubelet/pods/a72c435c-feac-4ef5-9dce-c6f23a90e7ae/volumes"
Oct 30 04:49:24.604510 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.604469    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc" podUID="44b5759d-ff2e-43c9-981f-5c04b1131ebf" containerName="payload-processing" containerID="cri-o://491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd" gracePeriod=25
Oct 30 04:49:24.745192 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.745158    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"]
Oct 30 04:49:24.745975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.745956    5072 cpu_manager.go:414] "containerMap: removing container" podUID="71ed81c9-f26d-4045-8180-835b41ddd212" containerName="payload-processing"
Oct 30 04:49:24.745975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.745973    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ed81c9-f26d-4045-8180-835b41ddd212" containerName="payload-processing"
Oct 30 04:49:24.746162 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.745992    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ae5e1716-636c-4077-9b7f-ae125f61ad14" containerName="revoke-keys"
Oct 30 04:49:24.746162 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.746002    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5e1716-636c-4077-9b7f-ae125f61ad14" containerName="revoke-keys"
Oct 30 04:49:24.746162 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.746043    5072 cpu_manager.go:414] "containerMap: removing container" podUID="fb2a3585-7fd4-4142-afcd-1c8976cff9b4" containerName="revoke-keys"
Oct 30 04:49:24.746162 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.746048    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2a3585-7fd4-4142-afcd-1c8976cff9b4" containerName="revoke-keys"
Oct 30 04:49:24.746162 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.746068    5072 cpu_manager.go:414] "containerMap: removing container" podUID="bb92f7ed-2b60-4ca7-93e4-072c0cb770b5" containerName="payload-processing"
Oct 30 04:49:24.746162 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.746091    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb92f7ed-2b60-4ca7-93e4-072c0cb770b5" containerName="payload-processing"
Oct 30 04:49:24.746469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.746222    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2a3585-7fd4-4142-afcd-1c8976cff9b4" containerName="revoke-keys"
Oct 30 04:49:24.746469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.746236    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb92f7ed-2b60-4ca7-93e4-072c0cb770b5" containerName="payload-processing"
Oct 30 04:49:24.746469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.746248    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ed81c9-f26d-4045-8180-835b41ddd212" containerName="payload-processing"
Oct 30 04:49:24.746469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.746259    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5e1716-636c-4077-9b7f-ae125f61ad14" containerName="revoke-keys"
Oct 30 04:49:24.750180 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.750157    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.771099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.769871    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0da867-beb3-412d-9af4-128b6c39dbdf-istiod-ca-cert\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.771099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.769949    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-credential-socket\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.771099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.770029    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-certs\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.771099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.770053    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-envoy\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.771099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.770178    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6glz\" (UniqueName: \"kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-kube-api-access-z6glz\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.771099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.770295    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-token\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.771099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.770349    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-podinfo\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.771099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.770399    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-data\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.771099 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.770442    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-socket\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.773842 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.772384    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b195d0-openshift-default-dockercfg-gkwn4\""
Oct 30 04:49:24.788740 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.788706    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"]
Oct 30 04:49:24.872151 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.872113    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0da867-beb3-412d-9af4-128b6c39dbdf-istiod-ca-cert\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.872371 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.872200    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-credential-socket\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.872371 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.872297    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-certs\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.872371 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.872322    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-envoy\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.872371 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.872355    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6glz\" (UniqueName: \"kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-kube-api-access-z6glz\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.872636 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.872476    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-token\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.872636 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.872532    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-podinfo\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.872636 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.872582    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-data\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.872636 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.872625    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-socket\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.873567 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.873158    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-socket\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.873825 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.873801    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0da867-beb3-412d-9af4-128b6c39dbdf-istiod-ca-cert\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.874060 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.874039    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-credential-socket\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.874345 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.874321    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-certs\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.875238 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.875096    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-data\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.877765 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.877718    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-envoy\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.879733 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.879633    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-podinfo\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.901505 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.901471    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6glz\" (UniqueName: \"kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-kube-api-access-z6glz\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.901657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.901640    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-token\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:24.969066 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:24.969041    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:49:25.069836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.069760    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:25.075105 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.075055    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-plugins-config-volume\") pod \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") "
Oct 30 04:49:25.075173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.075142    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-provider-credentials\") pod \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") "
Oct 30 04:49:25.075229 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.075212    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-routing-overlay-volume\") pod \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") "
Oct 30 04:49:25.075378 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.075364    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-kube-api-access-dzvs8\" (UniqueName: \"kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-kube-api-access-dzvs8\") pod \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\" (UID: \"44b5759d-ff2e-43c9-981f-5c04b1131ebf\") "
Oct 30 04:49:25.075465 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.075395    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-plugins-config-volume" pod "44b5759d-ff2e-43c9-981f-5c04b1131ebf" (UID: "44b5759d-ff2e-43c9-981f-5c04b1131ebf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:25.075589 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.075568    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-routing-overlay-volume" pod "44b5759d-ff2e-43c9-981f-5c04b1131ebf" (UID: "44b5759d-ff2e-43c9-981f-5c04b1131ebf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:25.075858 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.075841    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:25.075915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.075864    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44b5759d-ff2e-43c9-981f-5c04b1131ebf-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:25.077581 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.077558    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-provider-credentials" pod "44b5759d-ff2e-43c9-981f-5c04b1131ebf" (UID: "44b5759d-ff2e-43c9-981f-5c04b1131ebf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:25.077644 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.077596    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-kube-api-access-dzvs8" pod "44b5759d-ff2e-43c9-981f-5c04b1131ebf" (UID: "44b5759d-ff2e-43c9-981f-5c04b1131ebf"). InnerVolumeSpecName "kube-api-access-dzvs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:25.184777 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.184739    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzvs8\" (UniqueName: \"kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-kube-api-access-dzvs8\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:25.184952 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.184778    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44b5759d-ff2e-43c9-981f-5c04b1131ebf-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:25.191195 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.186613    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"]
Oct 30 04:49:25.191195 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.187337    5072 cpu_manager.go:414] "containerMap: removing container" podUID="44b5759d-ff2e-43c9-981f-5c04b1131ebf" containerName="payload-processing"
Oct 30 04:49:25.191195 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.187352    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b5759d-ff2e-43c9-981f-5c04b1131ebf" containerName="payload-processing"
Oct 30 04:49:25.191195 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.187496    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b5759d-ff2e-43c9-981f-5c04b1131ebf" containerName="payload-processing"
Oct 30 04:49:25.192864 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.192846    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.226455 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.226425    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"]
Oct 30 04:49:25.285876 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.285819    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.286137 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.285879    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.286209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.286189    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krld5\" (UniqueName: \"kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-kube-api-access-krld5\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.286259 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.286245    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.389945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.387898    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krld5\" (UniqueName: \"kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-kube-api-access-krld5\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.389945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.387954    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.389945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.388114    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.389945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.388161    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.389945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.389008    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.389945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.389780    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.397413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.395861    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"]
Oct 30 04:49:25.400553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.400512    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:49:25.400773 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.400749    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:49:25.400945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.400925    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:49:25.416499 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.416463    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krld5\" (UniqueName: \"kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-kube-api-access-krld5\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.427571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.425849    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.432484 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.432453    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"]
Oct 30 04:49:25.440438 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.440366    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:25.461021 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.459979    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:49:25.461021 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.460840    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:49:25.469891 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.468832    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"]
Oct 30 04:49:25.489724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.489689    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-882ee0b6-wmlt9\" (UID: \"db39036c-9f85-4b27-b099-abec63d0cd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:25.489918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.489769    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-882ee0b6-wmlt9\" (UID: \"db39036c-9f85-4b27-b099-abec63d0cd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:25.489918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.489901    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvrvz\" (UniqueName: \"kubernetes.io/projected/db39036c-9f85-4b27-b099-abec63d0cd54-kube-api-access-nvrvz\") pod \"maas-api-revoke-sub-882ee0b6-wmlt9\" (UID: \"db39036c-9f85-4b27-b099-abec63d0cd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:25.524791 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.524766    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:25.592298 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.592171    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrvz\" (UniqueName: \"kubernetes.io/projected/db39036c-9f85-4b27-b099-abec63d0cd54-kube-api-access-nvrvz\") pod \"maas-api-revoke-sub-882ee0b6-wmlt9\" (UID: \"db39036c-9f85-4b27-b099-abec63d0cd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:25.592445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.592324    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-882ee0b6-wmlt9\" (UID: \"db39036c-9f85-4b27-b099-abec63d0cd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:25.592445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.592365    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-882ee0b6-wmlt9\" (UID: \"db39036c-9f85-4b27-b099-abec63d0cd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:25.593315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.593266    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-882ee0b6-wmlt9\" (UID: \"db39036c-9f85-4b27-b099-abec63d0cd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:25.595325 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.595301    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-882ee0b6-wmlt9\" (UID: \"db39036c-9f85-4b27-b099-abec63d0cd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:25.616940 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.616852    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrvz\" (UniqueName: \"kubernetes.io/projected/db39036c-9f85-4b27-b099-abec63d0cd54-kube-api-access-nvrvz\") pod \"maas-api-revoke-sub-882ee0b6-wmlt9\" (UID: \"db39036c-9f85-4b27-b099-abec63d0cd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:25.650789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.650711    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"]
Oct 30 04:49:25.679304 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.679262    5072 generic.go:356] "Generic (PLEG): container finished" podID="44b5759d-ff2e-43c9-981f-5c04b1131ebf" containerID="491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd" exitCode=0
Oct 30 04:49:25.679476 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.679446    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc" event={"ID":"44b5759d-ff2e-43c9-981f-5c04b1131ebf","Type":"ContainerDied","Data":"491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd"}
Oct 30 04:49:25.679531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.679514    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc" event={"ID":"44b5759d-ff2e-43c9-981f-5c04b1131ebf","Type":"ContainerDied","Data":"b473e7de971fb7ef438a12520e2e7d9eab82707fea28c458b9d8dd97f80c102a"}
Oct 30 04:49:25.679601 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.679546    5072 scope.go:122] "RemoveContainer" containerID="491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd"
Oct 30 04:49:25.679736 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.679719    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"
Oct 30 04:49:25.706571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.705587    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf" event={"ID":"9d0da867-beb3-412d-9af4-128b6c39dbdf","Type":"ContainerStarted","Data":"fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514"}
Oct 30 04:49:25.706571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.705658    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf" event={"ID":"9d0da867-beb3-412d-9af4-128b6c39dbdf","Type":"ContainerStarted","Data":"d9b3da27a2b2f8ca8d6ba4b6107eeb4511cc0c005e476d846346e25badeb4555"}
Oct 30 04:49:25.724772 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.724749    5072 scope.go:122] "RemoveContainer" containerID="491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd"
Oct 30 04:49:25.725144 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:49:25.725111    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd\": container with ID starting with 491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd not found: ID does not exist" containerID="491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd"
Oct 30 04:49:25.725265 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.725155    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd"} err="failed to get container status \"491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd\": rpc error: code = NotFound desc = could not find container \"491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd\": container with ID starting with 491ec2be8c18ca90c5b736531558e7d30db3e469684bb13efb992a7c366469cd not found: ID does not exist"
Oct 30 04:49:25.819146 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.819057    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf" podStartSLOduration=1.819036197 podStartE2EDuration="1.819036197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:25.780286969 +0000 UTC m=+3145.521710058" watchObservedRunningTime="2026-10-30 04:49:25.819036197 +0000 UTC m=+3145.560459295"
Oct 30 04:49:25.824214 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.824187    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"]
Oct 30 04:49:25.824354 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.824316    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:25.830027 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.830002    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9d69tkc"]
Oct 30 04:49:25.840476 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:25.840451    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"]
Oct 30 04:49:25.842674 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:25.842635    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71810f06_aa27_40b1_b0a5_5dcbacd059b4.slice/crio-2a93bea7c2715aaab64816c2699c15568e84ea2be285d16bec57946ec99515b7 WatchSource:0}: Error finding container 2a93bea7c2715aaab64816c2699c15568e84ea2be285d16bec57946ec99515b7: Status 404 returned error can't find the container with id 2a93bea7c2715aaab64816c2699c15568e84ea2be285d16bec57946ec99515b7
Oct 30 04:49:26.070656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:26.070618    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:26.072234 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:26.072203    5072 patch_prober.go:29] interesting pod/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 04:49:26.072369 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:26.072263    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf" podUID="9d0da867-beb3-412d-9af4-128b6c39dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 04:49:26.155138 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:26.155110    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"]
Oct 30 04:49:26.156241 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:26.156206    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb39036c_9f85_4b27_b099_abec63d0cd54.slice/crio-5941b4d059e7ca5a66f55d8a844604a02399939d08706b7a8011588e09f2b184 WatchSource:0}: Error finding container 5941b4d059e7ca5a66f55d8a844604a02399939d08706b7a8011588e09f2b184: Status 404 returned error can't find the container with id 5941b4d059e7ca5a66f55d8a844604a02399939d08706b7a8011588e09f2b184
Oct 30 04:49:26.410666 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:26.410602    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b5759d-ff2e-43c9-981f-5c04b1131ebf" path="/var/lib/kubelet/pods/44b5759d-ff2e-43c9-981f-5c04b1131ebf/volumes"
Oct 30 04:49:26.717828 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:26.717794    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9" event={"ID":"db39036c-9f85-4b27-b099-abec63d0cd54","Type":"ContainerStarted","Data":"a5d888638b2ceaaf7f5a09bcd51e7e073c8af8f26e34f33931c332f3108199ba"}
Oct 30 04:49:26.717995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:26.717850    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9" event={"ID":"db39036c-9f85-4b27-b099-abec63d0cd54","Type":"ContainerStarted","Data":"5941b4d059e7ca5a66f55d8a844604a02399939d08706b7a8011588e09f2b184"}
Oct 30 04:49:26.719605 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:26.719575    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f" event={"ID":"71810f06-aa27-40b1-b0a5-5dcbacd059b4","Type":"ContainerStarted","Data":"8ca351c34415a233d834c04009f58d58ddb3bb81fecd89b3923ed56e27c790e9"}
Oct 30 04:49:26.719710 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:26.719630    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f" event={"ID":"71810f06-aa27-40b1-b0a5-5dcbacd059b4","Type":"ContainerStarted","Data":"2a93bea7c2715aaab64816c2699c15568e84ea2be285d16bec57946ec99515b7"}
Oct 30 04:49:26.719854 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:26.719837    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:26.789101 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:26.789022    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9" podStartSLOduration=1.789002404 podStartE2EDuration="1.789002404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:26.786379006 +0000 UTC m=+3146.527802104" watchObservedRunningTime="2026-10-30 04:49:26.789002404 +0000 UTC m=+3146.530425504"
Oct 30 04:49:26.839824 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:26.839777    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f" podStartSLOduration=1.839764473 podStartE2EDuration="1.839764473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:26.838112503 +0000 UTC m=+3146.579535602" watchObservedRunningTime="2026-10-30 04:49:26.839764473 +0000 UTC m=+3146.581187605"
Oct 30 04:49:27.075272 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.075176    5072 patch_prober.go:29] interesting pod/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 04:49:27.075841 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.075810    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf" podUID="9d0da867-beb3-412d-9af4-128b6c39dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 04:49:27.146161 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.146118    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj" podUID="62e30a5b-5ae8-4efb-8459-c0e094851980" containerName="payload-processing" containerID="cri-o://6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11" gracePeriod=25
Oct 30 04:49:27.479103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.478281    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:49:27.628453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.628361    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-kube-api-access-n95vn\" (UniqueName: \"kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-kube-api-access-n95vn\") pod \"62e30a5b-5ae8-4efb-8459-c0e094851980\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") "
Oct 30 04:49:27.628633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.628506    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-routing-overlay-volume\") pod \"62e30a5b-5ae8-4efb-8459-c0e094851980\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") "
Oct 30 04:49:27.628633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.628537    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-plugins-config-volume\") pod \"62e30a5b-5ae8-4efb-8459-c0e094851980\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") "
Oct 30 04:49:27.628633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.628599    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-provider-credentials\") pod \"62e30a5b-5ae8-4efb-8459-c0e094851980\" (UID: \"62e30a5b-5ae8-4efb-8459-c0e094851980\") "
Oct 30 04:49:27.628950 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.628919    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-routing-overlay-volume" pod "62e30a5b-5ae8-4efb-8459-c0e094851980" (UID: "62e30a5b-5ae8-4efb-8459-c0e094851980"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:27.629056 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.628951    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-plugins-config-volume" pod "62e30a5b-5ae8-4efb-8459-c0e094851980" (UID: "62e30a5b-5ae8-4efb-8459-c0e094851980"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:27.629244 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.629222    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:27.629297 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.629250    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e30a5b-5ae8-4efb-8459-c0e094851980-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:27.630866 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.630840    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-kube-api-access-n95vn" pod "62e30a5b-5ae8-4efb-8459-c0e094851980" (UID: "62e30a5b-5ae8-4efb-8459-c0e094851980"). InnerVolumeSpecName "kube-api-access-n95vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:27.630984 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.630903    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-provider-credentials" pod "62e30a5b-5ae8-4efb-8459-c0e094851980" (UID: "62e30a5b-5ae8-4efb-8459-c0e094851980"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:27.738252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.730659    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n95vn\" (UniqueName: \"kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-kube-api-access-n95vn\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:27.738252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.730696    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62e30a5b-5ae8-4efb-8459-c0e094851980-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:27.738252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.731769    5072 generic.go:356] "Generic (PLEG): container finished" podID="62e30a5b-5ae8-4efb-8459-c0e094851980" containerID="6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11" exitCode=0
Oct 30 04:49:27.738252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.732034    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj" event={"ID":"62e30a5b-5ae8-4efb-8459-c0e094851980","Type":"ContainerDied","Data":"6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11"}
Oct 30 04:49:27.738252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.732112    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj" event={"ID":"62e30a5b-5ae8-4efb-8459-c0e094851980","Type":"ContainerDied","Data":"5ae43966f2626096de5478e5df9099cea5ebebffe74e688e831cc5eace6b1d2b"}
Oct 30 04:49:27.738252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.732146    5072 scope.go:122] "RemoveContainer" containerID="6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11"
Oct 30 04:49:27.738252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.732313    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"
Oct 30 04:49:27.738252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.736658    5072 generic.go:356] "Generic (PLEG): container finished" podID="db39036c-9f85-4b27-b099-abec63d0cd54" containerID="a5d888638b2ceaaf7f5a09bcd51e7e073c8af8f26e34f33931c332f3108199ba" exitCode=0
Oct 30 04:49:27.738252 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.737841    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9" event={"ID":"db39036c-9f85-4b27-b099-abec63d0cd54","Type":"ContainerDied","Data":"a5d888638b2ceaaf7f5a09bcd51e7e073c8af8f26e34f33931c332f3108199ba"}
Oct 30 04:49:27.753189 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.753169    5072 scope.go:122] "RemoveContainer" containerID="6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11"
Oct 30 04:49:27.753550 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:49:27.753529    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11\": container with ID starting with 6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11 not found: ID does not exist" containerID="6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11"
Oct 30 04:49:27.753637 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.753557    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11"} err="failed to get container status \"6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11\": rpc error: code = NotFound desc = could not find container \"6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11\": container with ID starting with 6343a6932e4a0a102c6ae0d224b33bb412b609c7bad2d4b9df5e6d24ae27dc11 not found: ID does not exist"
Oct 30 04:49:27.838872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.838837    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"]
Oct 30 04:49:27.845123 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:27.845066    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-6666zwdfj"]
Oct 30 04:49:28.071396 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:28.071363    5072 patch_prober.go:29] interesting pod/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 04:49:28.071636 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:28.071597    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf" podUID="9d0da867-beb3-412d-9af4-128b6c39dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 04:49:28.408936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:28.408859    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e30a5b-5ae8-4efb-8459-c0e094851980" path="/var/lib/kubelet/pods/62e30a5b-5ae8-4efb-8459-c0e094851980/volumes"
Oct 30 04:49:28.900067 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:28.900029    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"]
Oct 30 04:49:28.901375 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:28.901350    5072 cpu_manager.go:414] "containerMap: removing container" podUID="62e30a5b-5ae8-4efb-8459-c0e094851980" containerName="payload-processing"
Oct 30 04:49:28.901469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:28.901378    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e30a5b-5ae8-4efb-8459-c0e094851980" containerName="payload-processing"
Oct 30 04:49:28.901575 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:28.901561    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e30a5b-5ae8-4efb-8459-c0e094851980" containerName="payload-processing"
Oct 30 04:49:28.909581 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:28.909557    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:28.917919 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:28.917894    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"]
Oct 30 04:49:29.001901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.001838    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:29.002117 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.001927    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"agc-cred-16d391637b1c\""
Oct 30 04:49:29.002199 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.002028    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"payload-processing-external-model-e2e--9d9c1ba6-dockercfg-59xxc\""
Oct 30 04:49:29.048680 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.048651    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.048857 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.048777    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.048857 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.048813    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.048991 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.048977    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkrgr\" (UniqueName: \"kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-kube-api-access-kkrgr\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.087811 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.087467    5072 patch_prober.go:29] interesting pod/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 04:49:29.087811 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.087527    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf" podUID="9d0da867-beb3-412d-9af4-128b6c39dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 04:49:29.149604 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.149566    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-xks-ca\") pod \"db39036c-9f85-4b27-b099-abec63d0cd54\" (UID: \"db39036c-9f85-4b27-b099-abec63d0cd54\") "
Oct 30 04:49:29.149802 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.149740    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-openshift-ca\") pod \"db39036c-9f85-4b27-b099-abec63d0cd54\" (UID: \"db39036c-9f85-4b27-b099-abec63d0cd54\") "
Oct 30 04:49:29.149802 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.149790    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db39036c-9f85-4b27-b099-abec63d0cd54-kube-api-access-nvrvz\" (UniqueName: \"kubernetes.io/projected/db39036c-9f85-4b27-b099-abec63d0cd54-kube-api-access-nvrvz\") pod \"db39036c-9f85-4b27-b099-abec63d0cd54\" (UID: \"db39036c-9f85-4b27-b099-abec63d0cd54\") "
Oct 30 04:49:29.150149 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.150121    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-openshift-ca" pod "db39036c-9f85-4b27-b099-abec63d0cd54" (UID: "db39036c-9f85-4b27-b099-abec63d0cd54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:29.150355 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.150157    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.150355 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.150201    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.150355 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.150263    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.150355 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.150371    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkrgr\" (UniqueName: \"kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-kube-api-access-kkrgr\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.150669 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.150465    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:29.150934 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.150908    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.151223 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.151198    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.152613 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.152585    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-xks-ca" pod "db39036c-9f85-4b27-b099-abec63d0cd54" (UID: "db39036c-9f85-4b27-b099-abec63d0cd54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:49:29.153060 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.153039    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db39036c-9f85-4b27-b099-abec63d0cd54-kube-api-access-nvrvz" pod "db39036c-9f85-4b27-b099-abec63d0cd54" (UID: "db39036c-9f85-4b27-b099-abec63d0cd54"). InnerVolumeSpecName "kube-api-access-nvrvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:29.153403 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.153386    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.195907 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.195867    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkrgr\" (UniqueName: \"kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-kube-api-access-kkrgr\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.252668 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.252597    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db39036c-9f85-4b27-b099-abec63d0cd54-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:29.252668 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.252626    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvrvz\" (UniqueName: \"kubernetes.io/projected/db39036c-9f85-4b27-b099-abec63d0cd54-kube-api-access-nvrvz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:29.304973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.304927    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:29.681154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.681108    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"]
Oct 30 04:49:29.691205 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:29.690324    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc751f43a_800a_4b5d_8084_b2f6a140b0a5.slice/crio-fb0d05411be72eeded213427dadfb66cbc9de48964c0ccadd6a1aaeba2d2a042 WatchSource:0}: Error finding container fb0d05411be72eeded213427dadfb66cbc9de48964c0ccadd6a1aaeba2d2a042: Status 404 returned error can't find the container with id fb0d05411be72eeded213427dadfb66cbc9de48964c0ccadd6a1aaeba2d2a042
Oct 30 04:49:29.783140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.782125    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9" event={"ID":"db39036c-9f85-4b27-b099-abec63d0cd54","Type":"ContainerDied","Data":"5941b4d059e7ca5a66f55d8a844604a02399939d08706b7a8011588e09f2b184"}
Oct 30 04:49:29.783140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.782190    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5941b4d059e7ca5a66f55d8a844604a02399939d08706b7a8011588e09f2b184"
Oct 30 04:49:29.783140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.782209    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"
Oct 30 04:49:29.785382 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:29.785353    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4" event={"ID":"c751f43a-800a-4b5d-8084-b2f6a140b0a5","Type":"ContainerStarted","Data":"fb0d05411be72eeded213427dadfb66cbc9de48964c0ccadd6a1aaeba2d2a042"}
Oct 30 04:49:30.075331 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.075296    5072 patch_prober.go:29] interesting pod/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 04:49:30.075590 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.075562    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf" podUID="9d0da867-beb3-412d-9af4-128b6c39dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 04:49:30.411129 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.411046    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"]
Oct 30 04:49:30.411772 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.411757    5072 cpu_manager.go:414] "containerMap: removing container" podUID="db39036c-9f85-4b27-b099-abec63d0cd54" containerName="revoke-keys"
Oct 30 04:49:30.411824 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.411774    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db39036c-9f85-4b27-b099-abec63d0cd54" containerName="revoke-keys"
Oct 30 04:49:30.411894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.411885    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="db39036c-9f85-4b27-b099-abec63d0cd54" containerName="revoke-keys"
Oct 30 04:49:30.415597 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.415575    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.429880 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.429854    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"]
Oct 30 04:49:30.437704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.437682    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"agc-cred-983e0efbe50a\""
Oct 30 04:49:30.492642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.491998    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.492642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.492231    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.492642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.492263    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2zs2\" (UniqueName: \"kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-kube-api-access-t2zs2\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.492642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.492279    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.593152 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.593116    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.593353 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.593332    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.593404 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.593375    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2zs2\" (UniqueName: \"kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-kube-api-access-t2zs2\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.593449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.593400    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.594151 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.594123    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.594151 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.594145    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.596370 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.596347    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.617007 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.616987    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2zs2\" (UniqueName: \"kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-kube-api-access-t2zs2\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.651584 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.651550    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6" podUID="26a81b73-3cb5-4f40-b91b-fb824821c9f2" containerName="payload-processing" containerID="cri-o://0496999182e2e4cbf1c3105f0b3d99a77c234a48f69ecb8cf2f63d6ef85484cd" gracePeriod=25
Oct 30 04:49:30.733854 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.733823    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:30.797682 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.797646    5072 generic.go:356] "Generic (PLEG): container finished" podID="26a81b73-3cb5-4f40-b91b-fb824821c9f2" containerID="0496999182e2e4cbf1c3105f0b3d99a77c234a48f69ecb8cf2f63d6ef85484cd" exitCode=0
Oct 30 04:49:30.797805 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.797751    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6" event={"ID":"26a81b73-3cb5-4f40-b91b-fb824821c9f2","Type":"ContainerDied","Data":"0496999182e2e4cbf1c3105f0b3d99a77c234a48f69ecb8cf2f63d6ef85484cd"}
Oct 30 04:49:30.800310 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.800278    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4" event={"ID":"c751f43a-800a-4b5d-8084-b2f6a140b0a5","Type":"ContainerStarted","Data":"0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a"}
Oct 30 04:49:30.800427 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.800365    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:30.843704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.843678    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"]
Oct 30 04:49:30.853958 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.853898    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4" podStartSLOduration=2.8538804559999997 podStartE2EDuration="2.853880456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:30.847996102 +0000 UTC m=+3150.589419211" watchObservedRunningTime="2026-10-30 04:49:30.853880456 +0000 UTC m=+3150.595303554"
Oct 30 04:49:30.893317 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:30.893229    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"]
Oct 30 04:49:31.079522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.079485    5072 patch_prober.go:29] interesting pod/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 04:49:31.079789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.079760    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf" podUID="9d0da867-beb3-412d-9af4-128b6c39dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 04:49:31.095472 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:31.095443    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d9c083_9be6_42fd_b059_a4a5791bc93d.slice/crio-128724e3382430e95ca8656660257b759ea0d90181ca432437cc076a6e63a73f WatchSource:0}: Error finding container 128724e3382430e95ca8656660257b759ea0d90181ca432437cc076a6e63a73f: Status 404 returned error can't find the container with id 128724e3382430e95ca8656660257b759ea0d90181ca432437cc076a6e63a73f
Oct 30 04:49:31.096905 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.096883    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"]
Oct 30 04:49:31.117044 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.117016    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:49:31.208006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.206991    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-plugins-config-volume\") pod \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") "
Oct 30 04:49:31.208006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.207068    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-kube-api-access-kwt4v\" (UniqueName: \"kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-kube-api-access-kwt4v\") pod \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") "
Oct 30 04:49:31.208006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.207245    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-provider-credentials\") pod \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") "
Oct 30 04:49:31.208006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.207319    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-routing-overlay-volume\") pod \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\" (UID: \"26a81b73-3cb5-4f40-b91b-fb824821c9f2\") "
Oct 30 04:49:31.208866 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.208538    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-routing-overlay-volume" pod "26a81b73-3cb5-4f40-b91b-fb824821c9f2" (UID: "26a81b73-3cb5-4f40-b91b-fb824821c9f2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:31.210027 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.210005    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-plugins-config-volume" pod "26a81b73-3cb5-4f40-b91b-fb824821c9f2" (UID: "26a81b73-3cb5-4f40-b91b-fb824821c9f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:31.212803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.212772    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-provider-credentials" pod "26a81b73-3cb5-4f40-b91b-fb824821c9f2" (UID: "26a81b73-3cb5-4f40-b91b-fb824821c9f2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:31.212967 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.212941    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-kube-api-access-kwt4v" pod "26a81b73-3cb5-4f40-b91b-fb824821c9f2" (UID: "26a81b73-3cb5-4f40-b91b-fb824821c9f2"). InnerVolumeSpecName "kube-api-access-kwt4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:31.308556 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.308520    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:31.308556 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.308550    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwt4v\" (UniqueName: \"kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-kube-api-access-kwt4v\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:31.308556 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.308561    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26a81b73-3cb5-4f40-b91b-fb824821c9f2-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:31.308794 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.308570    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26a81b73-3cb5-4f40-b91b-fb824821c9f2-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:31.813531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.813442    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"
Oct 30 04:49:31.813531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.813467    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6" event={"ID":"26a81b73-3cb5-4f40-b91b-fb824821c9f2","Type":"ContainerDied","Data":"0ef6368f955a88a0444d6c5888b53907708013c0dbc8464c55af570bdd70a681"}
Oct 30 04:49:31.814329 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.813801    5072 scope.go:122] "RemoveContainer" containerID="0496999182e2e4cbf1c3105f0b3d99a77c234a48f69ecb8cf2f63d6ef85484cd"
Oct 30 04:49:31.817827 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.817799    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp" event={"ID":"b6d9c083-9be6-42fd-b059-a4a5791bc93d","Type":"ContainerStarted","Data":"41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d"}
Oct 30 04:49:31.817989 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.817886    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp" event={"ID":"b6d9c083-9be6-42fd-b059-a4a5791bc93d","Type":"ContainerStarted","Data":"128724e3382430e95ca8656660257b759ea0d90181ca432437cc076a6e63a73f"}
Oct 30 04:49:31.817989 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.817938    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:31.867446 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:49:31.867411    5072 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a81b73_3cb5_4f40_b91b_fb824821c9f2.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 04:49:31.872596 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.872570    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"]
Oct 30 04:49:31.878668 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.878640    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-58842j7s6"]
Oct 30 04:49:31.936895 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:31.936823    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp" podStartSLOduration=1.936804114 podStartE2EDuration="1.936804114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:31.931581078 +0000 UTC m=+3151.673004176" watchObservedRunningTime="2026-10-30 04:49:31.936804114 +0000 UTC m=+3151.678227212"
Oct 30 04:49:32.117485 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:32.117401    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:32.118001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:32.117974    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:32.118933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:32.118914    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 04:49:32.411047 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:32.410974    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a81b73-3cb5-4f40-b91b-fb824821c9f2" path="/var/lib/kubelet/pods/26a81b73-3cb5-4f40-b91b-fb824821c9f2/volumes"
Oct 30 04:49:34.053437 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:34.053399    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"]
Oct 30 04:49:34.065849 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:34.065824    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7f76dd-wpc72"]
Oct 30 04:49:34.410247 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:34.410161    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82602269-b23c-4bbc-a5ec-c3d32d148f1b" path="/var/lib/kubelet/pods/82602269-b23c-4bbc-a5ec-c3d32d148f1b/volumes"
Oct 30 04:49:35.051505 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:35.051471    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"]
Oct 30 04:49:35.054979 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:35.054954    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ebd708-n5h2r"]
Oct 30 04:49:35.845008 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:35.844970    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq" podUID="faae7ed3-be0f-445a-9b4e-fe7674637de2" containerName="payload-processing" containerID="cri-o://0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96" gracePeriod=25
Oct 30 04:49:35.894672 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:35.894639    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr" podUID="bc684e7d-f6c1-4011-8632-810ba45a1970" containerName="payload-pre-processing" containerID="cri-o://744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43" gracePeriod=25
Oct 30 04:49:36.243471 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.243145    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"
Oct 30 04:49:36.249205 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.249181    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"
Oct 30 04:49:36.374560 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.374513    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc684e7d-f6c1-4011-8632-810ba45a1970-kube-api-access-6vdxr\" (UniqueName: \"kubernetes.io/projected/bc684e7d-f6c1-4011-8632-810ba45a1970-kube-api-access-6vdxr\") pod \"bc684e7d-f6c1-4011-8632-810ba45a1970\" (UID: \"bc684e7d-f6c1-4011-8632-810ba45a1970\") "
Oct 30 04:49:36.374738 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.374604    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faae7ed3-be0f-445a-9b4e-fe7674637de2-kube-api-access-2lc7q\" (UniqueName: \"kubernetes.io/projected/faae7ed3-be0f-445a-9b4e-fe7674637de2-kube-api-access-2lc7q\") pod \"faae7ed3-be0f-445a-9b4e-fe7674637de2\" (UID: \"faae7ed3-be0f-445a-9b4e-fe7674637de2\") "
Oct 30 04:49:36.374738 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.374674    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faae7ed3-be0f-445a-9b4e-fe7674637de2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faae7ed3-be0f-445a-9b4e-fe7674637de2-plugins-config-volume\") pod \"faae7ed3-be0f-445a-9b4e-fe7674637de2\" (UID: \"faae7ed3-be0f-445a-9b4e-fe7674637de2\") "
Oct 30 04:49:36.374842 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.374735    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc684e7d-f6c1-4011-8632-810ba45a1970-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc684e7d-f6c1-4011-8632-810ba45a1970-plugins-config-volume\") pod \"bc684e7d-f6c1-4011-8632-810ba45a1970\" (UID: \"bc684e7d-f6c1-4011-8632-810ba45a1970\") "
Oct 30 04:49:36.375872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.375840    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faae7ed3-be0f-445a-9b4e-fe7674637de2-plugins-config-volume" pod "faae7ed3-be0f-445a-9b4e-fe7674637de2" (UID: "faae7ed3-be0f-445a-9b4e-fe7674637de2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:36.375978 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.375850    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc684e7d-f6c1-4011-8632-810ba45a1970-plugins-config-volume" pod "bc684e7d-f6c1-4011-8632-810ba45a1970" (UID: "bc684e7d-f6c1-4011-8632-810ba45a1970"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:36.379450 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.379351    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc684e7d-f6c1-4011-8632-810ba45a1970-kube-api-access-6vdxr" pod "bc684e7d-f6c1-4011-8632-810ba45a1970" (UID: "bc684e7d-f6c1-4011-8632-810ba45a1970"). InnerVolumeSpecName "kube-api-access-6vdxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:36.379804 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.379767    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faae7ed3-be0f-445a-9b4e-fe7674637de2-kube-api-access-2lc7q" pod "faae7ed3-be0f-445a-9b4e-fe7674637de2" (UID: "faae7ed3-be0f-445a-9b4e-fe7674637de2"). InnerVolumeSpecName "kube-api-access-2lc7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:36.425689 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.425658    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a718c06-ccfe-4fcb-913f-29f6b21d6aa0" path="/var/lib/kubelet/pods/7a718c06-ccfe-4fcb-913f-29f6b21d6aa0/volumes"
Oct 30 04:49:36.480116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.480056    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vdxr\" (UniqueName: \"kubernetes.io/projected/bc684e7d-f6c1-4011-8632-810ba45a1970-kube-api-access-6vdxr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:36.480116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.480117    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lc7q\" (UniqueName: \"kubernetes.io/projected/faae7ed3-be0f-445a-9b4e-fe7674637de2-kube-api-access-2lc7q\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:36.480387 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.480134    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faae7ed3-be0f-445a-9b4e-fe7674637de2-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:36.480387 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.480148    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc684e7d-f6c1-4011-8632-810ba45a1970-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:36.696836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.696806    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"]
Oct 30 04:49:36.699760 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.699737    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:36.700688 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.700667    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"]
Oct 30 04:49:36.862765 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.862721    5072 generic.go:356] "Generic (PLEG): container finished" podID="bc684e7d-f6c1-4011-8632-810ba45a1970" containerID="744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43" exitCode=0
Oct 30 04:49:36.862933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.862792    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"
Oct 30 04:49:36.862933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.862826    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr" event={"ID":"bc684e7d-f6c1-4011-8632-810ba45a1970","Type":"ContainerDied","Data":"744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43"}
Oct 30 04:49:36.862933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.862892    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr" event={"ID":"bc684e7d-f6c1-4011-8632-810ba45a1970","Type":"ContainerDied","Data":"2ac9fe9af645f894c7ce56a37d67098970f01cb632b23644243d7bd4c6518c79"}
Oct 30 04:49:36.862933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.862925    5072 scope.go:122] "RemoveContainer" containerID="744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43"
Oct 30 04:49:36.864732 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.864664    5072 generic.go:356] "Generic (PLEG): container finished" podID="faae7ed3-be0f-445a-9b4e-fe7674637de2" containerID="0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96" exitCode=0
Oct 30 04:49:36.864836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.864753    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq" event={"ID":"faae7ed3-be0f-445a-9b4e-fe7674637de2","Type":"ContainerDied","Data":"0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96"}
Oct 30 04:49:36.864836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.864788    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"
Oct 30 04:49:36.864836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.864801    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq" event={"ID":"faae7ed3-be0f-445a-9b4e-fe7674637de2","Type":"ContainerDied","Data":"750b15f2d673824b92d536feee0e770e3e6033b489a7bf0d2be29ae9d42ae3c1"}
Oct 30 04:49:36.875918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.875900    5072 scope.go:122] "RemoveContainer" containerID="744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43"
Oct 30 04:49:36.876247 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:49:36.876220    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43\": container with ID starting with 744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43 not found: ID does not exist" containerID="744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43"
Oct 30 04:49:36.876343 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.876252    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43"} err="failed to get container status \"744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43\": rpc error: code = NotFound desc = could not find container \"744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43\": container with ID starting with 744232d8832ac0b8fe656f3c4957aa38021a6e56cb15092330aa3a1099224c43 not found: ID does not exist"
Oct 30 04:49:36.876343 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.876271    5072 scope.go:122] "RemoveContainer" containerID="0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96"
Oct 30 04:49:36.887369 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.887351    5072 scope.go:122] "RemoveContainer" containerID="0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96"
Oct 30 04:49:36.887652 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:49:36.887635    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96\": container with ID starting with 0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96 not found: ID does not exist" containerID="0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96"
Oct 30 04:49:36.887713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.887658    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96"} err="failed to get container status \"0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96\": rpc error: code = NotFound desc = could not find container \"0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96\": container with ID starting with 0f64293f4e5190c005c6c378f22317cf0feb0d67b3ea475d4946762abafa6d96 not found: ID does not exist"
Oct 30 04:49:36.919028 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.919007    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"]
Oct 30 04:49:36.931089 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.931059    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-721c85-76b87c7df5-kl8rr"]
Oct 30 04:49:36.974678 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.974622    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"]
Oct 30 04:49:36.979241 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:36.979210    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-721c85-7fb8654d7b-tjrcq"]
Oct 30 04:49:38.410656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:38.410619    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc684e7d-f6c1-4011-8632-810ba45a1970" path="/var/lib/kubelet/pods/bc684e7d-f6c1-4011-8632-810ba45a1970/volumes"
Oct 30 04:49:38.411277 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:38.411257    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faae7ed3-be0f-445a-9b4e-fe7674637de2" path="/var/lib/kubelet/pods/faae7ed3-be0f-445a-9b4e-fe7674637de2/volumes"
Oct 30 04:49:41.643600 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.643558    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"]
Oct 30 04:49:41.644571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.644545    5072 cpu_manager.go:414] "containerMap: removing container" podUID="bc684e7d-f6c1-4011-8632-810ba45a1970" containerName="payload-pre-processing"
Oct 30 04:49:41.644571 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.644570    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc684e7d-f6c1-4011-8632-810ba45a1970" containerName="payload-pre-processing"
Oct 30 04:49:41.644733 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.644634    5072 cpu_manager.go:414] "containerMap: removing container" podUID="26a81b73-3cb5-4f40-b91b-fb824821c9f2" containerName="payload-processing"
Oct 30 04:49:41.644733 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.644644    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a81b73-3cb5-4f40-b91b-fb824821c9f2" containerName="payload-processing"
Oct 30 04:49:41.644733 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.644662    5072 cpu_manager.go:414] "containerMap: removing container" podUID="faae7ed3-be0f-445a-9b4e-fe7674637de2" containerName="payload-processing"
Oct 30 04:49:41.644733 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.644668    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faae7ed3-be0f-445a-9b4e-fe7674637de2" containerName="payload-processing"
Oct 30 04:49:41.644935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.644837    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc684e7d-f6c1-4011-8632-810ba45a1970" containerName="payload-pre-processing"
Oct 30 04:49:41.644935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.644860    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a81b73-3cb5-4f40-b91b-fb824821c9f2" containerName="payload-processing"
Oct 30 04:49:41.644935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.644870    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="faae7ed3-be0f-445a-9b4e-fe7674637de2" containerName="payload-processing"
Oct 30 04:49:41.650256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.650234    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.666984 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.666953    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"kube-root-ca.crt\""
Oct 30 04:49:41.668726 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.668704    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"]
Oct 30 04:49:41.677402 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.677377    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"openshift-service-ca.crt\""
Oct 30 04:49:41.677402 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.677386    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"payload-processing-external-model-e2e--e5d834c2-dockercfg-s8hvz\""
Oct 30 04:49:41.677658 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.677624    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c0e964\""
Oct 30 04:49:41.677658 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.677644    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"agc-cred-637f0a1096f5\""
Oct 30 04:49:41.677801 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.677692    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"routing-overlay\""
Oct 30 04:49:41.697997 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.697949    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f" podUID="71810f06-aa27-40b1-b0a5-5dcbacd059b4" containerName="payload-processing" containerID="cri-o://8ca351c34415a233d834c04009f58d58ddb3bb81fecd89b3923ed56e27c790e9" gracePeriod=25
Oct 30 04:49:41.701568 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.701529    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw" podUID="b821e187-e48f-4ce4-92fd-b0185252d142" containerName="payload-processing" containerID="cri-o://b8aae1c1a3f4f4dc634ce9a91eb04ed1f7b17e2a58ba0a36f9fec4d014d0cc2c" gracePeriod=25
Oct 30 04:49:41.742624 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.742598    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.742713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.742642    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.742713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.742660    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bphbw\" (UniqueName: \"kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-kube-api-access-bphbw\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.742948 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.742922    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.754551 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:49:41.754519    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:49:41.832871 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.831665    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:41.854641 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.853527    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.854641 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.853775    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.854641 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.853812    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bphbw\" (UniqueName: \"kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-kube-api-access-bphbw\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.854641 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.854315    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.854977 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.854908    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.855224 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.855195    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.857861 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.857835    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.878395 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.878190    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bphbw\" (UniqueName: \"kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-kube-api-access-bphbw\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:41.926053 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.925753    5072 generic.go:356] "Generic (PLEG): container finished" podID="b821e187-e48f-4ce4-92fd-b0185252d142" containerID="b8aae1c1a3f4f4dc634ce9a91eb04ed1f7b17e2a58ba0a36f9fec4d014d0cc2c" exitCode=0
Oct 30 04:49:41.926053 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.925961    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw" event={"ID":"b821e187-e48f-4ce4-92fd-b0185252d142","Type":"ContainerDied","Data":"b8aae1c1a3f4f4dc634ce9a91eb04ed1f7b17e2a58ba0a36f9fec4d014d0cc2c"}
Oct 30 04:49:41.928011 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.927903    5072 generic.go:356] "Generic (PLEG): container finished" podID="71810f06-aa27-40b1-b0a5-5dcbacd059b4" containerID="8ca351c34415a233d834c04009f58d58ddb3bb81fecd89b3923ed56e27c790e9" exitCode=0
Oct 30 04:49:41.928011 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.927940    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f" event={"ID":"71810f06-aa27-40b1-b0a5-5dcbacd059b4","Type":"ContainerDied","Data":"8ca351c34415a233d834c04009f58d58ddb3bb81fecd89b3923ed56e27c790e9"}
Oct 30 04:49:41.966131 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:41.966051    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:42.021650 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.020790    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:49:42.057492 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.057468    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-routing-overlay-volume\") pod \"b821e187-e48f-4ce4-92fd-b0185252d142\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") "
Oct 30 04:49:42.057908 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.057893    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-plugins-config-volume\") pod \"b821e187-e48f-4ce4-92fd-b0185252d142\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") "
Oct 30 04:49:42.058383 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.058370    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-provider-credentials\") pod \"b821e187-e48f-4ce4-92fd-b0185252d142\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") "
Oct 30 04:49:42.058530 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.058514    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-kube-api-access-4ldmk\" (UniqueName: \"kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-kube-api-access-4ldmk\") pod \"b821e187-e48f-4ce4-92fd-b0185252d142\" (UID: \"b821e187-e48f-4ce4-92fd-b0185252d142\") "
Oct 30 04:49:42.058901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.057851    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-routing-overlay-volume" pod "b821e187-e48f-4ce4-92fd-b0185252d142" (UID: "b821e187-e48f-4ce4-92fd-b0185252d142"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:42.058901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.058306    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-plugins-config-volume" pod "b821e187-e48f-4ce4-92fd-b0185252d142" (UID: "b821e187-e48f-4ce4-92fd-b0185252d142"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:42.059619 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.059587    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:42.059700 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.059625    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b821e187-e48f-4ce4-92fd-b0185252d142-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:42.079416 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.074402    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-provider-credentials" pod "b821e187-e48f-4ce4-92fd-b0185252d142" (UID: "b821e187-e48f-4ce4-92fd-b0185252d142"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:42.079502 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.079471    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:42.079770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.079727    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-kube-api-access-4ldmk" pod "b821e187-e48f-4ce4-92fd-b0185252d142" (UID: "b821e187-e48f-4ce4-92fd-b0185252d142"). InnerVolumeSpecName "kube-api-access-4ldmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:42.160452 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.160399    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-provider-credentials\") pod \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") "
Oct 30 04:49:42.160621 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.160596    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-routing-overlay-volume\") pod \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") "
Oct 30 04:49:42.160685 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.160645    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-kube-api-access-krld5\" (UniqueName: \"kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-kube-api-access-krld5\") pod \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") "
Oct 30 04:49:42.160782 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.160721    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-plugins-config-volume\") pod \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\" (UID: \"71810f06-aa27-40b1-b0a5-5dcbacd059b4\") "
Oct 30 04:49:42.161018 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.160992    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-routing-overlay-volume" pod "71810f06-aa27-40b1-b0a5-5dcbacd059b4" (UID: "71810f06-aa27-40b1-b0a5-5dcbacd059b4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:42.161299 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.161281    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:42.161397 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.161301    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:42.161397 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.161312    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ldmk\" (UniqueName: \"kubernetes.io/projected/b821e187-e48f-4ce4-92fd-b0185252d142-kube-api-access-4ldmk\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:42.161525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.161412    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-plugins-config-volume" pod "71810f06-aa27-40b1-b0a5-5dcbacd059b4" (UID: "71810f06-aa27-40b1-b0a5-5dcbacd059b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:42.167089 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.167043    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-kube-api-access-krld5" pod "71810f06-aa27-40b1-b0a5-5dcbacd059b4" (UID: "71810f06-aa27-40b1-b0a5-5dcbacd059b4"). InnerVolumeSpecName "kube-api-access-krld5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:42.168335 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.168313    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-provider-credentials" pod "71810f06-aa27-40b1-b0a5-5dcbacd059b4" (UID: "71810f06-aa27-40b1-b0a5-5dcbacd059b4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:42.240393 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.240359    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"]
Oct 30 04:49:42.241064 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.241048    5072 cpu_manager.go:414] "containerMap: removing container" podUID="71810f06-aa27-40b1-b0a5-5dcbacd059b4" containerName="payload-processing"
Oct 30 04:49:42.241175 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.241066    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71810f06-aa27-40b1-b0a5-5dcbacd059b4" containerName="payload-processing"
Oct 30 04:49:42.241175 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.241152    5072 cpu_manager.go:414] "containerMap: removing container" podUID="b821e187-e48f-4ce4-92fd-b0185252d142" containerName="payload-processing"
Oct 30 04:49:42.241175 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.241159    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b821e187-e48f-4ce4-92fd-b0185252d142" containerName="payload-processing"
Oct 30 04:49:42.241291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.241281    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b821e187-e48f-4ce4-92fd-b0185252d142" containerName="payload-processing"
Oct 30 04:49:42.241327 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.241300    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="71810f06-aa27-40b1-b0a5-5dcbacd059b4" containerName="payload-processing"
Oct 30 04:49:42.245253 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.245233    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.263143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.263114    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71810f06-aa27-40b1-b0a5-5dcbacd059b4-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:42.263143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.263144    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:42.263316 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.263157    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krld5\" (UniqueName: \"kubernetes.io/projected/71810f06-aa27-40b1-b0a5-5dcbacd059b4-kube-api-access-krld5\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:42.267006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.266981    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"agc-cred-6eb495e9b647\""
Oct 30 04:49:42.267699 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.267670    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"]
Oct 30 04:49:42.316126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.311246    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"]
Oct 30 04:49:42.316845 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:42.316820    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fa16e7_dcd7_4735_88b6_e66107d6e63a.slice/crio-a8899ee1dc7594f0824ed9d6d1649b5b1baf022262014c08728aef0bf25e31d7 WatchSource:0}: Error finding container a8899ee1dc7594f0824ed9d6d1649b5b1baf022262014c08728aef0bf25e31d7: Status 404 returned error can't find the container with id a8899ee1dc7594f0824ed9d6d1649b5b1baf022262014c08728aef0bf25e31d7
Oct 30 04:49:42.364479 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.364447    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.364606 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.364494    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.364606 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.364563    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.364747 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.364724    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkgdp\" (UniqueName: \"kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-kube-api-access-gkgdp\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.466533 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.466488    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.466704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.466669    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkgdp\" (UniqueName: \"kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-kube-api-access-gkgdp\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.466841 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.466815    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.467022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.466855    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.467412 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.467390    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.467522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.467497    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.469557 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.469537    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.491034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.490956    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkgdp\" (UniqueName: \"kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-kube-api-access-gkgdp\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.560494 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.560456    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:42.830782 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.830717    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:49:42.868531 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:42.868502    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5aa91f_09e1_45e5_abe7_f0c9575c2139.slice/crio-33df459567f9e1ccb996ed8347a7968d96df7114ac68ca63d90de520b07ca63e WatchSource:0}: Error finding container 33df459567f9e1ccb996ed8347a7968d96df7114ac68ca63d90de520b07ca63e: Status 404 returned error can't find the container with id 33df459567f9e1ccb996ed8347a7968d96df7114ac68ca63d90de520b07ca63e
Oct 30 04:49:42.871011 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.870980    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"]
Oct 30 04:49:42.911344 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.911300    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"]
Oct 30 04:49:42.941308 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.939518    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw" event={"ID":"b821e187-e48f-4ce4-92fd-b0185252d142","Type":"ContainerDied","Data":"558602961bdf903ec7939be6c7d04d790934b6e1b630b61d6f4534ec571934e6"}
Oct 30 04:49:42.941308 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.939587    5072 scope.go:122] "RemoveContainer" containerID="b8aae1c1a3f4f4dc634ce9a91eb04ed1f7b17e2a58ba0a36f9fec4d014d0cc2c"
Oct 30 04:49:42.941308 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.939701    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"
Oct 30 04:49:42.942864 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.942830    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4" event={"ID":"22fa16e7-dcd7-4735-88b6-e66107d6e63a","Type":"ContainerStarted","Data":"50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b"}
Oct 30 04:49:42.942984 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.942888    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4" event={"ID":"22fa16e7-dcd7-4735-88b6-e66107d6e63a","Type":"ContainerStarted","Data":"a8899ee1dc7594f0824ed9d6d1649b5b1baf022262014c08728aef0bf25e31d7"}
Oct 30 04:49:42.943967 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.943671    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:42.947400 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.947377    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6" event={"ID":"4a5aa91f-09e1-45e5-abe7-f0c9575c2139","Type":"ContainerStarted","Data":"33df459567f9e1ccb996ed8347a7968d96df7114ac68ca63d90de520b07ca63e"}
Oct 30 04:49:42.949111 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.949065    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f" event={"ID":"71810f06-aa27-40b1-b0a5-5dcbacd059b4","Type":"ContainerDied","Data":"2a93bea7c2715aaab64816c2699c15568e84ea2be285d16bec57946ec99515b7"}
Oct 30 04:49:42.949229 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.949180    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"
Oct 30 04:49:42.955222 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.955205    5072 scope.go:122] "RemoveContainer" containerID="8ca351c34415a233d834c04009f58d58ddb3bb81fecd89b3923ed56e27c790e9"
Oct 30 04:49:42.993870 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:42.993845    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"]
Oct 30 04:49:43.000027 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.000001    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-7699xhpkw"]
Oct 30 04:49:43.035619 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.035588    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"]
Oct 30 04:49:43.041760 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.041737    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-5d85tbg6f"]
Oct 30 04:49:43.084770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.084683    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4" podStartSLOduration=2.084669074 podStartE2EDuration="2.084669074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:43.083194747 +0000 UTC m=+3162.824617847" watchObservedRunningTime="2026-10-30 04:49:43.084669074 +0000 UTC m=+3162.826092171"
Oct 30 04:49:43.366882 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.366800    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"]
Oct 30 04:49:43.372627 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.372599    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.396052 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.395998    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"]
Oct 30 04:49:43.401109 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.398051    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c0d6e7-openshift-default-dockercfg-5rg25\""
Oct 30 04:49:43.478269 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.478237    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-credential-socket\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.478269 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.478272    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-certs\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.478537 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.478297    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-socket\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.478537 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.478412    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04291e68-092a-4c76-bb56-52ce38990484-istio-podinfo\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.478537 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.478463    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vpb\" (UniqueName: \"kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-kube-api-access-c8vpb\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.478749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.478543    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04291e68-092a-4c76-bb56-52ce38990484-istiod-ca-cert\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.478749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.478594    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-data\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.478749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.478644    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-istio-token\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.478749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.478717    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-envoy\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.579892 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.579857    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-istio-token\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.580126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.579911    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-envoy\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.580126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.579984    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-credential-socket\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.580126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.580014    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-certs\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.580126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.580045    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-socket\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.580366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.580143    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04291e68-092a-4c76-bb56-52ce38990484-istio-podinfo\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.580366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.580226    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vpb\" (UniqueName: \"kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-kube-api-access-c8vpb\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.580366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.580286    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04291e68-092a-4c76-bb56-52ce38990484-istiod-ca-cert\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.580366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.580337    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-data\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.580587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.580478    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-credential-socket\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.580587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.580541    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-socket\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.580749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.580694    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-data\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.580869 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.580789    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-certs\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.581347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.581326    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04291e68-092a-4c76-bb56-52ce38990484-istiod-ca-cert\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.582590 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.582568    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-envoy\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.583360 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.583339    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04291e68-092a-4c76-bb56-52ce38990484-istio-podinfo\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.604164 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.604137    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-istio-token\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.604319 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.604297    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vpb\" (UniqueName: \"kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-kube-api-access-c8vpb\") pod \"e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") " pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.689326 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.689296    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:43.962851 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:43.962770    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6" event={"ID":"4a5aa91f-09e1-45e5-abe7-f0c9575c2139","Type":"ContainerStarted","Data":"3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d"}
Oct 30 04:49:44.007546 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:44.007484    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6" podStartSLOduration=2.007470147 podStartE2EDuration="2.007470147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:44.00567286 +0000 UTC m=+3163.747095960" watchObservedRunningTime="2026-10-30 04:49:44.007470147 +0000 UTC m=+3163.748893291"
Oct 30 04:49:44.033592 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:44.033561    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04291e68_092a_4c76_bb56_52ce38990484.slice/crio-1c9c8ca862f157ba55c926b3635f7bf3838ac3c83c26daa770ae371bf7ab984d WatchSource:0}: Error finding container 1c9c8ca862f157ba55c926b3635f7bf3838ac3c83c26daa770ae371bf7ab984d: Status 404 returned error can't find the container with id 1c9c8ca862f157ba55c926b3635f7bf3838ac3c83c26daa770ae371bf7ab984d
Oct 30 04:49:44.037184 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:44.035970    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"]
Oct 30 04:49:44.044462 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:44.044403    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:49:44.044717 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:44.044686    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:49:44.044909 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:44.044888    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:49:44.416992 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:44.416919    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71810f06-aa27-40b1-b0a5-5dcbacd059b4" path="/var/lib/kubelet/pods/71810f06-aa27-40b1-b0a5-5dcbacd059b4/volumes"
Oct 30 04:49:44.417390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:44.417373    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b821e187-e48f-4ce4-92fd-b0185252d142" path="/var/lib/kubelet/pods/b821e187-e48f-4ce4-92fd-b0185252d142/volumes"
Oct 30 04:49:44.971734 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:44.971697    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" event={"ID":"04291e68-092a-4c76-bb56-52ce38990484","Type":"ContainerStarted","Data":"cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec"}
Oct 30 04:49:44.972241 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:44.971751    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" event={"ID":"04291e68-092a-4c76-bb56-52ce38990484","Type":"ContainerStarted","Data":"1c9c8ca862f157ba55c926b3635f7bf3838ac3c83c26daa770ae371bf7ab984d"}
Oct 30 04:49:44.972241 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:44.971858    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:45.033057 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:45.033010    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" podStartSLOduration=2.03299688 podStartE2EDuration="2.03299688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:45.018458122 +0000 UTC m=+3164.759881227" watchObservedRunningTime="2026-10-30 04:49:45.03299688 +0000 UTC m=+3164.774419974"
Oct 30 04:49:45.689634 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:45.689607    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:45.695654 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:45.695631    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:45.979787 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:45.979706    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:45.980770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:45.980750    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:49:47.530996 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.530966    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"]
Oct 30 04:49:47.536453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.536428    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.562220 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.562194    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"]
Oct 30 04:49:47.569903 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.569858    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 04:49:47.569903 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.569868    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 04:49:47.569903 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.569868    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 04:49:47.570231 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.569968    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2vrdc\""
Oct 30 04:49:47.627618 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.627580    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7kbp\" (UniqueName: \"kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-kube-api-access-x7kbp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.627771 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.627703    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.627771 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.627733    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.627849 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.627812    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd75cac-bcad-4d17-93fb-817d5576a32e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.627849 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.627836    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.627982 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.627961    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.628046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.628032    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.729634 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.729596    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.729634 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.729635    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7kbp\" (UniqueName: \"kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-kube-api-access-x7kbp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.729886 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.729705    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.729886 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.729727    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.729886 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.729803    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd75cac-bcad-4d17-93fb-817d5576a32e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.729886 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.729836    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.730143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.729927    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.730584 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.730557    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.732794 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.732761    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.732916 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.732812    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd75cac-bcad-4d17-93fb-817d5576a32e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.732975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.732908    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.733016 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.733005    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.753953 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.753927    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7kbp\" (UniqueName: \"kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-kube-api-access-x7kbp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.754124 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.754104    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9hb8\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.850038 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.849952    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:47.912371 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:47.912332    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4" podUID="c751f43a-800a-4b5d-8084-b2f6a140b0a5" containerName="payload-processing" containerID="cri-o://0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a" gracePeriod=25
Oct 30 04:49:48.156271 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.156244    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:48.175487 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:49:48.175448    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd75cac_bcad_4d17_93fb_817d5576a32e.slice/crio-e13e009a3644a38b5145bdd100fad38e064a235b3a1229bcb27916ccc616ab96 WatchSource:0}: Error finding container e13e009a3644a38b5145bdd100fad38e064a235b3a1229bcb27916ccc616ab96: Status 404 returned error can't find the container with id e13e009a3644a38b5145bdd100fad38e064a235b3a1229bcb27916ccc616ab96
Oct 30 04:49:48.176525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.176426    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"]
Oct 30 04:49:48.179068 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.179025    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:49:48.179193 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.179128    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:49:48.236013 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.235987    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-kube-api-access-kkrgr\" (UniqueName: \"kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-kube-api-access-kkrgr\") pod \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") "
Oct 30 04:49:48.236150 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.236071    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-provider-credentials\") pod \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") "
Oct 30 04:49:48.236225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.236178    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-plugins-config-volume\") pod \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") "
Oct 30 04:49:48.236331 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.236311    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-routing-overlay-volume\") pod \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\" (UID: \"c751f43a-800a-4b5d-8084-b2f6a140b0a5\") "
Oct 30 04:49:48.236765 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.236672    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-plugins-config-volume" pod "c751f43a-800a-4b5d-8084-b2f6a140b0a5" (UID: "c751f43a-800a-4b5d-8084-b2f6a140b0a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:48.236765 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.236730    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-routing-overlay-volume" pod "c751f43a-800a-4b5d-8084-b2f6a140b0a5" (UID: "c751f43a-800a-4b5d-8084-b2f6a140b0a5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:48.236949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.236833    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:48.236949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.236846    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c751f43a-800a-4b5d-8084-b2f6a140b0a5-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:48.239188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.238832    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-kube-api-access-kkrgr" pod "c751f43a-800a-4b5d-8084-b2f6a140b0a5" (UID: "c751f43a-800a-4b5d-8084-b2f6a140b0a5"). InnerVolumeSpecName "kube-api-access-kkrgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:48.239188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.238877    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-provider-credentials" pod "c751f43a-800a-4b5d-8084-b2f6a140b0a5" (UID: "c751f43a-800a-4b5d-8084-b2f6a140b0a5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:48.338633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.338598    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkrgr\" (UniqueName: \"kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-kube-api-access-kkrgr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:48.338633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:48.338631    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c751f43a-800a-4b5d-8084-b2f6a140b0a5-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:49:49.025435 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.025323    5072 generic.go:356] "Generic (PLEG): container finished" podID="c751f43a-800a-4b5d-8084-b2f6a140b0a5" containerID="0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a" exitCode=0
Oct 30 04:49:49.025928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.025432    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4" event={"ID":"c751f43a-800a-4b5d-8084-b2f6a140b0a5","Type":"ContainerDied","Data":"0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a"}
Oct 30 04:49:49.025928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.025486    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4" event={"ID":"c751f43a-800a-4b5d-8084-b2f6a140b0a5","Type":"ContainerDied","Data":"fb0d05411be72eeded213427dadfb66cbc9de48964c0ccadd6a1aaeba2d2a042"}
Oct 30 04:49:49.025928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.025519    5072 scope.go:122] "RemoveContainer" containerID="0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a"
Oct 30 04:49:49.025928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.025698    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"
Oct 30 04:49:49.032490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.032458    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8" event={"ID":"7cd75cac-bcad-4d17-93fb-817d5576a32e","Type":"ContainerStarted","Data":"6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528"}
Oct 30 04:49:49.032614 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.032516    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8" event={"ID":"7cd75cac-bcad-4d17-93fb-817d5576a32e","Type":"ContainerStarted","Data":"e13e009a3644a38b5145bdd100fad38e064a235b3a1229bcb27916ccc616ab96"}
Oct 30 04:49:49.033358 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.033332    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:49.035730 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.035484    5072 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-b9hb8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:49:49.035730 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.035533    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8" podUID="7cd75cac-bcad-4d17-93fb-817d5576a32e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:49:49.085795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.084651    5072 scope.go:122] "RemoveContainer" containerID="0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a"
Oct 30 04:49:49.086410 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:49:49.086338    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a\": container with ID starting with 0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a not found: ID does not exist" containerID="0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a"
Oct 30 04:49:49.086410 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.086392    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a"} err="failed to get container status \"0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a\": rpc error: code = NotFound desc = could not find container \"0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a\": container with ID starting with 0d598efe285bf1eed1ec3c9549060289628ce47e506a1a4d3e26789ad2a2a80a not found: ID does not exist"
Oct 30 04:49:49.118249 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.118199    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8" podStartSLOduration=2.1181833 podStartE2EDuration="2.1181833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:49.115467895 +0000 UTC m=+3168.856890996" watchObservedRunningTime="2026-10-30 04:49:49.1181833 +0000 UTC m=+3168.859606408"
Oct 30 04:49:49.157221 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.157192    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"]
Oct 30 04:49:49.162282 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:49.162256    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfjtfw4"]
Oct 30 04:49:50.042483 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:50.042448    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 04:49:50.410389 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:50.410314    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c751f43a-800a-4b5d-8084-b2f6a140b0a5" path="/var/lib/kubelet/pods/c751f43a-800a-4b5d-8084-b2f6a140b0a5/volumes"
Oct 30 04:49:54.975460 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:54.975432    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:49:55.982684 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:55.982653    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:49:56.084171 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:49:56.084135    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"]
Oct 30 04:50:01.085390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.085351    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4" podUID="22fa16e7-dcd7-4735-88b6-e66107d6e63a" containerName="payload-processing" containerID="cri-o://50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b" gracePeriod=25
Oct 30 04:50:01.362743 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.362715    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:50:01.512784 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.512745    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-provider-credentials\") pod \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") "
Oct 30 04:50:01.513041 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.513021    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-routing-overlay-volume\") pod \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") "
Oct 30 04:50:01.513179 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.513058    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-plugins-config-volume\") pod \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") "
Oct 30 04:50:01.513179 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.513129    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-kube-api-access-bphbw\" (UniqueName: \"kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-kube-api-access-bphbw\") pod \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\" (UID: \"22fa16e7-dcd7-4735-88b6-e66107d6e63a\") "
Oct 30 04:50:01.513454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.513431    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-routing-overlay-volume" pod "22fa16e7-dcd7-4735-88b6-e66107d6e63a" (UID: "22fa16e7-dcd7-4735-88b6-e66107d6e63a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:50:01.513531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.513487    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-plugins-config-volume" pod "22fa16e7-dcd7-4735-88b6-e66107d6e63a" (UID: "22fa16e7-dcd7-4735-88b6-e66107d6e63a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:50:01.513686 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.513667    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:01.513741 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.513693    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fa16e7-dcd7-4735-88b6-e66107d6e63a-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:01.515630 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.515606    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-kube-api-access-bphbw" pod "22fa16e7-dcd7-4735-88b6-e66107d6e63a" (UID: "22fa16e7-dcd7-4735-88b6-e66107d6e63a"). InnerVolumeSpecName "kube-api-access-bphbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:50:01.515719 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.515676    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-provider-credentials" pod "22fa16e7-dcd7-4735-88b6-e66107d6e63a" (UID: "22fa16e7-dcd7-4735-88b6-e66107d6e63a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:50:01.538393 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.538369    5072 scope.go:122] "RemoveContainer" containerID="81f3fdf758c4a05499a7761c9c4d665b60806f62b8a4392cae05eaa6a7b240cf"
Oct 30 04:50:01.549957 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.549939    5072 scope.go:122] "RemoveContainer" containerID="43debf8a10f4b97ca035dc106637f204dab33f8131b1fcfff9378e5a282b4ea9"
Oct 30 04:50:01.615147 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.615125    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:01.615147 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:01.615148    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bphbw\" (UniqueName: \"kubernetes.io/projected/22fa16e7-dcd7-4735-88b6-e66107d6e63a-kube-api-access-bphbw\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:02.138157 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:02.138105    5072 generic.go:356] "Generic (PLEG): container finished" podID="22fa16e7-dcd7-4735-88b6-e66107d6e63a" containerID="50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b" exitCode=0
Oct 30 04:50:02.138157 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:02.138156    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"
Oct 30 04:50:02.138692 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:02.138181    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4" event={"ID":"22fa16e7-dcd7-4735-88b6-e66107d6e63a","Type":"ContainerDied","Data":"50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b"}
Oct 30 04:50:02.138692 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:02.138247    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4" event={"ID":"22fa16e7-dcd7-4735-88b6-e66107d6e63a","Type":"ContainerDied","Data":"a8899ee1dc7594f0824ed9d6d1649b5b1baf022262014c08728aef0bf25e31d7"}
Oct 30 04:50:02.138692 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:02.138288    5072 scope.go:122] "RemoveContainer" containerID="50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b"
Oct 30 04:50:02.153407 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:02.153387    5072 scope.go:122] "RemoveContainer" containerID="50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b"
Oct 30 04:50:02.153710 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:02.153687    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b\": container with ID starting with 50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b not found: ID does not exist" containerID="50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b"
Oct 30 04:50:02.153771 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:02.153720    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b"} err="failed to get container status \"50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b\": rpc error: code = NotFound desc = could not find container \"50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b\": container with ID starting with 50ec9539f51d96f4e7d8a3878753e49ea0361589eade5d1bbd0dc7ad6943432b not found: ID does not exist"
Oct 30 04:50:02.193865 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:02.193841    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"]
Oct 30 04:50:02.206650 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:02.206621    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-6ffd74lm4"]
Oct 30 04:50:02.411228 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:02.411139    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fa16e7-dcd7-4735-88b6-e66107d6e63a" path="/var/lib/kubelet/pods/22fa16e7-dcd7-4735-88b6-e66107d6e63a/volumes"
Oct 30 04:50:11.752842 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:11.752763    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:50:11.810819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.810789    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"]
Oct 30 04:50:11.811575 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.811555    5072 cpu_manager.go:414] "containerMap: removing container" podUID="c751f43a-800a-4b5d-8084-b2f6a140b0a5" containerName="payload-processing"
Oct 30 04:50:11.811575 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.811573    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c751f43a-800a-4b5d-8084-b2f6a140b0a5" containerName="payload-processing"
Oct 30 04:50:11.811775 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.811594    5072 cpu_manager.go:414] "containerMap: removing container" podUID="22fa16e7-dcd7-4735-88b6-e66107d6e63a" containerName="payload-processing"
Oct 30 04:50:11.811775 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.811598    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fa16e7-dcd7-4735-88b6-e66107d6e63a" containerName="payload-processing"
Oct 30 04:50:11.811775 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.811721    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c751f43a-800a-4b5d-8084-b2f6a140b0a5" containerName="payload-processing"
Oct 30 04:50:11.811775 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.811735    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fa16e7-dcd7-4735-88b6-e66107d6e63a" containerName="payload-processing"
Oct 30 04:50:11.817097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.817064    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:11.833963 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.833930    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"]
Oct 30 04:50:11.841811 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.841789    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:50:11.841921 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.841887    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:50:11.937745 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.937714    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdhs\" (UniqueName: \"kubernetes.io/projected/fe57e396-3a5b-491e-bc59-b70fd9487ae5-kube-api-access-qfdhs\") pod \"maas-api-revoke-sub-d29de31b-m54hr\" (UID: \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:11.937885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.937791    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d29de31b-m54hr\" (UID: \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:11.937885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:11.937850    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d29de31b-m54hr\" (UID: \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:12.039347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:12.039257    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d29de31b-m54hr\" (UID: \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:12.039347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:12.039303    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d29de31b-m54hr\" (UID: \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:12.039577 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:12.039484    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdhs\" (UniqueName: \"kubernetes.io/projected/fe57e396-3a5b-491e-bc59-b70fd9487ae5-kube-api-access-qfdhs\") pod \"maas-api-revoke-sub-d29de31b-m54hr\" (UID: \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:12.040025 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:12.040006    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d29de31b-m54hr\" (UID: \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:12.041877 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:12.041858    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d29de31b-m54hr\" (UID: \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:12.063344 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:12.063314    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdhs\" (UniqueName: \"kubernetes.io/projected/fe57e396-3a5b-491e-bc59-b70fd9487ae5-kube-api-access-qfdhs\") pod \"maas-api-revoke-sub-d29de31b-m54hr\" (UID: \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:12.132333 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:12.132304    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:12.439241 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:50:12.439208    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe57e396_3a5b_491e_bc59_b70fd9487ae5.slice/crio-e6f0d2e6d00cd145fb8391f7cd2b299f3a86a75e6a12656496b1be40a2b5cd32 WatchSource:0}: Error finding container e6f0d2e6d00cd145fb8391f7cd2b299f3a86a75e6a12656496b1be40a2b5cd32: Status 404 returned error can't find the container with id e6f0d2e6d00cd145fb8391f7cd2b299f3a86a75e6a12656496b1be40a2b5cd32
Oct 30 04:50:12.439940 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:12.439917    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"]
Oct 30 04:50:13.225646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:13.225608    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr" event={"ID":"fe57e396-3a5b-491e-bc59-b70fd9487ae5","Type":"ContainerStarted","Data":"cd516aa05d94ea9e5f571b25053d550b60e01b0affcf762522e934a854dfa72c"}
Oct 30 04:50:13.226025 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:13.225663    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr" event={"ID":"fe57e396-3a5b-491e-bc59-b70fd9487ae5","Type":"ContainerStarted","Data":"e6f0d2e6d00cd145fb8391f7cd2b299f3a86a75e6a12656496b1be40a2b5cd32"}
Oct 30 04:50:13.267728 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:13.267666    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr" podStartSLOduration=2.267649599 podStartE2EDuration="2.267649599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:50:13.266993167 +0000 UTC m=+3193.008416265" watchObservedRunningTime="2026-10-30 04:50:13.267649599 +0000 UTC m=+3193.009072697"
Oct 30 04:50:14.234300 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:14.234265    5072 generic.go:356] "Generic (PLEG): container finished" podID="fe57e396-3a5b-491e-bc59-b70fd9487ae5" containerID="cd516aa05d94ea9e5f571b25053d550b60e01b0affcf762522e934a854dfa72c" exitCode=0
Oct 30 04:50:14.234785 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:14.234353    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr" event={"ID":"fe57e396-3a5b-491e-bc59-b70fd9487ae5","Type":"ContainerDied","Data":"cd516aa05d94ea9e5f571b25053d550b60e01b0affcf762522e934a854dfa72c"}
Oct 30 04:50:15.398105 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:15.398064    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:15.490156 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:15.490126    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-openshift-ca\") pod \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\" (UID: \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\") "
Oct 30 04:50:15.490348 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:15.490277    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-xks-ca\") pod \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\" (UID: \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\") "
Oct 30 04:50:15.490348 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:15.490303    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe57e396-3a5b-491e-bc59-b70fd9487ae5-kube-api-access-qfdhs\" (UniqueName: \"kubernetes.io/projected/fe57e396-3a5b-491e-bc59-b70fd9487ae5-kube-api-access-qfdhs\") pod \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\" (UID: \"fe57e396-3a5b-491e-bc59-b70fd9487ae5\") "
Oct 30 04:50:15.490549 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:15.490526    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-openshift-ca" pod "fe57e396-3a5b-491e-bc59-b70fd9487ae5" (UID: "fe57e396-3a5b-491e-bc59-b70fd9487ae5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:50:15.490720 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:15.490704    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:15.492666 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:15.492601    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-xks-ca" pod "fe57e396-3a5b-491e-bc59-b70fd9487ae5" (UID: "fe57e396-3a5b-491e-bc59-b70fd9487ae5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:50:15.492666 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:15.492649    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe57e396-3a5b-491e-bc59-b70fd9487ae5-kube-api-access-qfdhs" pod "fe57e396-3a5b-491e-bc59-b70fd9487ae5" (UID: "fe57e396-3a5b-491e-bc59-b70fd9487ae5"). InnerVolumeSpecName "kube-api-access-qfdhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:50:15.591473 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:15.591443    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe57e396-3a5b-491e-bc59-b70fd9487ae5-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:15.591473 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:15.591473    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdhs\" (UniqueName: \"kubernetes.io/projected/fe57e396-3a5b-491e-bc59-b70fd9487ae5-kube-api-access-qfdhs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:16.250783 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:16.250753    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"
Oct 30 04:50:16.250783 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:16.250761    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr" event={"ID":"fe57e396-3a5b-491e-bc59-b70fd9487ae5","Type":"ContainerDied","Data":"e6f0d2e6d00cd145fb8391f7cd2b299f3a86a75e6a12656496b1be40a2b5cd32"}
Oct 30 04:50:16.250999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:16.250838    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6f0d2e6d00cd145fb8391f7cd2b299f3a86a75e6a12656496b1be40a2b5cd32"
Oct 30 04:50:23.893979 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:23.893943    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"]
Oct 30 04:50:23.895868 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:23.895837    5072 cpu_manager.go:414] "containerMap: removing container" podUID="fe57e396-3a5b-491e-bc59-b70fd9487ae5" containerName="revoke-keys"
Oct 30 04:50:23.896057 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:23.896037    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe57e396-3a5b-491e-bc59-b70fd9487ae5" containerName="revoke-keys"
Oct 30 04:50:23.896475 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:23.896458    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe57e396-3a5b-491e-bc59-b70fd9487ae5" containerName="revoke-keys"
Oct 30 04:50:23.911472 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:23.911439    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"]
Oct 30 04:50:23.914161 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:23.911591    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:23.928198 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:23.928156    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:50:23.928361 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:23.928336    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:50:23.990420 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:23.990388    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c45f5ab0-rqrwj\" (UID: \"f1b12704-1363-4bec-867f-e3923d9b0fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:23.990583 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:23.990425    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfvph\" (UniqueName: \"kubernetes.io/projected/f1b12704-1363-4bec-867f-e3923d9b0fad-kube-api-access-qfvph\") pod \"maas-api-revoke-sub-c45f5ab0-rqrwj\" (UID: \"f1b12704-1363-4bec-867f-e3923d9b0fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:23.990583 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:23.990514    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c45f5ab0-rqrwj\" (UID: \"f1b12704-1363-4bec-867f-e3923d9b0fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:24.091983 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:24.091953    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfvph\" (UniqueName: \"kubernetes.io/projected/f1b12704-1363-4bec-867f-e3923d9b0fad-kube-api-access-qfvph\") pod \"maas-api-revoke-sub-c45f5ab0-rqrwj\" (UID: \"f1b12704-1363-4bec-867f-e3923d9b0fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:24.092162 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:24.091994    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c45f5ab0-rqrwj\" (UID: \"f1b12704-1363-4bec-867f-e3923d9b0fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:24.092240 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:24.092184    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c45f5ab0-rqrwj\" (UID: \"f1b12704-1363-4bec-867f-e3923d9b0fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:24.092876 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:24.092847    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c45f5ab0-rqrwj\" (UID: \"f1b12704-1363-4bec-867f-e3923d9b0fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:24.094687 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:24.094662    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c45f5ab0-rqrwj\" (UID: \"f1b12704-1363-4bec-867f-e3923d9b0fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:24.115488 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:24.115462    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfvph\" (UniqueName: \"kubernetes.io/projected/f1b12704-1363-4bec-867f-e3923d9b0fad-kube-api-access-qfvph\") pod \"maas-api-revoke-sub-c45f5ab0-rqrwj\" (UID: \"f1b12704-1363-4bec-867f-e3923d9b0fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:24.235989 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:24.235968    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:24.556982 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:24.556864    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"]
Oct 30 04:50:24.564450 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:50:24.564412    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b12704_1363_4bec_867f_e3923d9b0fad.slice/crio-23bfc274ac67c36ed594ade865c5a85aba99f2b03ab8258074c375df65bd2f4a WatchSource:0}: Error finding container 23bfc274ac67c36ed594ade865c5a85aba99f2b03ab8258074c375df65bd2f4a: Status 404 returned error can't find the container with id 23bfc274ac67c36ed594ade865c5a85aba99f2b03ab8258074c375df65bd2f4a
Oct 30 04:50:25.333359 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:25.333256    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj" event={"ID":"f1b12704-1363-4bec-867f-e3923d9b0fad","Type":"ContainerStarted","Data":"9d298192a58bd1a54a64b4658dd3f4f7ab9d41a2e1d9b86c7bb1daa417defe42"}
Oct 30 04:50:25.333359 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:25.333326    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj" event={"ID":"f1b12704-1363-4bec-867f-e3923d9b0fad","Type":"ContainerStarted","Data":"23bfc274ac67c36ed594ade865c5a85aba99f2b03ab8258074c375df65bd2f4a"}
Oct 30 04:50:25.377680 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:25.377611    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj" podStartSLOduration=2.377588976 podStartE2EDuration="2.377588976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:50:25.374616175 +0000 UTC m=+3205.116039273" watchObservedRunningTime="2026-10-30 04:50:25.377588976 +0000 UTC m=+3205.119012075"
Oct 30 04:50:26.343336 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:26.342961    5072 generic.go:356] "Generic (PLEG): container finished" podID="f1b12704-1363-4bec-867f-e3923d9b0fad" containerID="9d298192a58bd1a54a64b4658dd3f4f7ab9d41a2e1d9b86c7bb1daa417defe42" exitCode=0
Oct 30 04:50:26.343336 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:26.343258    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj" event={"ID":"f1b12704-1363-4bec-867f-e3923d9b0fad","Type":"ContainerDied","Data":"9d298192a58bd1a54a64b4658dd3f4f7ab9d41a2e1d9b86c7bb1daa417defe42"}
Oct 30 04:50:27.528351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:27.528326    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:27.539217 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:27.539187    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-503804/agc-cred-e68b135ffabe: secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:27.539217 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:27.539215    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh: secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:27.539452 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:27.539271    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials podName:1c31590c-7aee-47d9-9e1e-1dd95994401a nodeName:}" failed. No retries permitted until 2026-10-30 04:50:28.039249378 +0000 UTC m=+3207.780672456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-503804-848d98lsh" (UID: "1c31590c-7aee-47d9-9e1e-1dd95994401a") : secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:27.640215 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:27.640131    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-openshift-ca\") pod \"f1b12704-1363-4bec-867f-e3923d9b0fad\" (UID: \"f1b12704-1363-4bec-867f-e3923d9b0fad\") "
Oct 30 04:50:27.640215 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:27.640189    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-xks-ca\") pod \"f1b12704-1363-4bec-867f-e3923d9b0fad\" (UID: \"f1b12704-1363-4bec-867f-e3923d9b0fad\") "
Oct 30 04:50:27.640470 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:27.640259    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b12704-1363-4bec-867f-e3923d9b0fad-kube-api-access-qfvph\" (UniqueName: \"kubernetes.io/projected/f1b12704-1363-4bec-867f-e3923d9b0fad-kube-api-access-qfvph\") pod \"f1b12704-1363-4bec-867f-e3923d9b0fad\" (UID: \"f1b12704-1363-4bec-867f-e3923d9b0fad\") "
Oct 30 04:50:27.640581 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:27.640556    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-openshift-ca" pod "f1b12704-1363-4bec-867f-e3923d9b0fad" (UID: "f1b12704-1363-4bec-867f-e3923d9b0fad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:50:27.640915 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:27.640896    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:27.642626 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:27.642603    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-xks-ca" pod "f1b12704-1363-4bec-867f-e3923d9b0fad" (UID: "f1b12704-1363-4bec-867f-e3923d9b0fad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:50:27.642677 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:27.642627    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b12704-1363-4bec-867f-e3923d9b0fad-kube-api-access-qfvph" pod "f1b12704-1363-4bec-867f-e3923d9b0fad" (UID: "f1b12704-1363-4bec-867f-e3923d9b0fad"). InnerVolumeSpecName "kube-api-access-qfvph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:50:27.742538 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:27.742500    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b12704-1363-4bec-867f-e3923d9b0fad-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:27.742538 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:27.742530    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfvph\" (UniqueName: \"kubernetes.io/projected/f1b12704-1363-4bec-867f-e3923d9b0fad-kube-api-access-qfvph\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:28.045834 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:28.045800    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-503804/agc-cred-e68b135ffabe: secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:28.045834 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:28.045830    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh: secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:28.046042 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:28.045884    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials podName:1c31590c-7aee-47d9-9e1e-1dd95994401a nodeName:}" failed. No retries permitted until 2026-10-30 04:50:29.045869859 +0000 UTC m=+3208.787292934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-503804-848d98lsh" (UID: "1c31590c-7aee-47d9-9e1e-1dd95994401a") : secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:28.359528 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:28.359443    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj" event={"ID":"f1b12704-1363-4bec-867f-e3923d9b0fad","Type":"ContainerDied","Data":"23bfc274ac67c36ed594ade865c5a85aba99f2b03ab8258074c375df65bd2f4a"}
Oct 30 04:50:28.359528 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:28.359479    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"
Oct 30 04:50:28.359528 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:28.359515    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23bfc274ac67c36ed594ade865c5a85aba99f2b03ab8258074c375df65bd2f4a"
Oct 30 04:50:29.058164 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:29.058132    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-503804/agc-cred-e68b135ffabe: secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:29.058164 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:29.058158    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh: secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:29.058633 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:29.058210    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials podName:1c31590c-7aee-47d9-9e1e-1dd95994401a nodeName:}" failed. No retries permitted until 2026-10-30 04:50:31.058196131 +0000 UTC m=+3210.799619231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-503804-848d98lsh" (UID: "1c31590c-7aee-47d9-9e1e-1dd95994401a") : secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:31.084034 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:31.083999    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-503804/agc-cred-e68b135ffabe: secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:31.084034 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:31.084025    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh: secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:31.084510 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:31.084110    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials podName:1c31590c-7aee-47d9-9e1e-1dd95994401a nodeName:}" failed. No retries permitted until 2026-10-30 04:50:35.084064527 +0000 UTC m=+3214.825487618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-503804-848d98lsh" (UID: "1c31590c-7aee-47d9-9e1e-1dd95994401a") : secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:35.132584 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:35.132547    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-503804/agc-cred-1b38359d0333: secret "agc-cred-1b38359d0333" not found
Oct 30 04:50:35.132584 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:35.132593    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-503804/agc-cred-e68b135ffabe: secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:35.133128 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:35.132608    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh: [secret "agc-cred-1b38359d0333" not found, secret "agc-cred-e68b135ffabe" not found]
Oct 30 04:50:35.133128 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:35.132691    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials podName:1c31590c-7aee-47d9-9e1e-1dd95994401a nodeName:}" failed. No retries permitted until 2026-10-30 04:50:43.132669767 +0000 UTC m=+3222.874092865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-503804-848d98lsh" (UID: "1c31590c-7aee-47d9-9e1e-1dd95994401a") : [secret "agc-cred-1b38359d0333" not found, secret "agc-cred-e68b135ffabe" not found]
Oct 30 04:50:35.851400 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:35.851361    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"]
Oct 30 04:50:35.852551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:35.852522    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f1b12704-1363-4bec-867f-e3923d9b0fad" containerName="revoke-keys"
Oct 30 04:50:35.852551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:35.852545    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b12704-1363-4bec-867f-e3923d9b0fad" containerName="revoke-keys"
Oct 30 04:50:35.852776 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:35.852716    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b12704-1363-4bec-867f-e3923d9b0fad" containerName="revoke-keys"
Oct 30 04:50:35.856446 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:35.856415    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:35.883268 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:35.883159    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:50:35.883268 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:35.883228    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:50:35.893653 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:35.893620    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"]
Oct 30 04:50:35.947512 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:35.947475    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-713164b5-5wxlk\" (UID: \"f5d68385-595f-4d22-b761-e2be6ce315e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:35.947708 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:35.947551    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-713164b5-5wxlk\" (UID: \"f5d68385-595f-4d22-b761-e2be6ce315e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:35.947708 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:35.947672    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95tdk\" (UniqueName: \"kubernetes.io/projected/f5d68385-595f-4d22-b761-e2be6ce315e4-kube-api-access-95tdk\") pod \"maas-api-revoke-sub-713164b5-5wxlk\" (UID: \"f5d68385-595f-4d22-b761-e2be6ce315e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:36.048925 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:36.048887    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-713164b5-5wxlk\" (UID: \"f5d68385-595f-4d22-b761-e2be6ce315e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:36.049174 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:36.049028    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95tdk\" (UniqueName: \"kubernetes.io/projected/f5d68385-595f-4d22-b761-e2be6ce315e4-kube-api-access-95tdk\") pod \"maas-api-revoke-sub-713164b5-5wxlk\" (UID: \"f5d68385-595f-4d22-b761-e2be6ce315e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:36.049174 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:36.049150    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-713164b5-5wxlk\" (UID: \"f5d68385-595f-4d22-b761-e2be6ce315e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:36.049786 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:36.049764    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-713164b5-5wxlk\" (UID: \"f5d68385-595f-4d22-b761-e2be6ce315e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:36.051576 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:36.051552    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-713164b5-5wxlk\" (UID: \"f5d68385-595f-4d22-b761-e2be6ce315e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:36.073587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:36.073559    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95tdk\" (UniqueName: \"kubernetes.io/projected/f5d68385-595f-4d22-b761-e2be6ce315e4-kube-api-access-95tdk\") pod \"maas-api-revoke-sub-713164b5-5wxlk\" (UID: \"f5d68385-595f-4d22-b761-e2be6ce315e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:36.187246 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:36.187221    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:36.498576 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:50:36.498542    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d68385_595f_4d22_b761_e2be6ce315e4.slice/crio-caab3a206d5e4bd81b9c219f700c2b0d0a7eb8859ed853e8045eae8356c5f127 WatchSource:0}: Error finding container caab3a206d5e4bd81b9c219f700c2b0d0a7eb8859ed853e8045eae8356c5f127: Status 404 returned error can't find the container with id caab3a206d5e4bd81b9c219f700c2b0d0a7eb8859ed853e8045eae8356c5f127
Oct 30 04:50:36.498905 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:36.498881    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"]
Oct 30 04:50:37.431203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:37.431101    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk" event={"ID":"f5d68385-595f-4d22-b761-e2be6ce315e4","Type":"ContainerStarted","Data":"6f24ea536ae585b7135d8cc5574749204fd6403f07684fe2e8626157fd6390fa"}
Oct 30 04:50:37.431203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:37.431180    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk" event={"ID":"f5d68385-595f-4d22-b761-e2be6ce315e4","Type":"ContainerStarted","Data":"caab3a206d5e4bd81b9c219f700c2b0d0a7eb8859ed853e8045eae8356c5f127"}
Oct 30 04:50:37.488615 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:37.488552    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk" podStartSLOduration=2.4885316189999998 podStartE2EDuration="2.488531619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:50:37.472159727 +0000 UTC m=+3217.213582826" watchObservedRunningTime="2026-10-30 04:50:37.488531619 +0000 UTC m=+3217.229954717"
Oct 30 04:50:38.439457 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:38.439425    5072 generic.go:356] "Generic (PLEG): container finished" podID="f5d68385-595f-4d22-b761-e2be6ce315e4" containerID="6f24ea536ae585b7135d8cc5574749204fd6403f07684fe2e8626157fd6390fa" exitCode=0
Oct 30 04:50:38.439912 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:38.439506    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk" event={"ID":"f5d68385-595f-4d22-b761-e2be6ce315e4","Type":"ContainerDied","Data":"6f24ea536ae585b7135d8cc5574749204fd6403f07684fe2e8626157fd6390fa"}
Oct 30 04:50:39.634067 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:39.634043    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:39.722496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:39.721666    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-openshift-ca\") pod \"f5d68385-595f-4d22-b761-e2be6ce315e4\" (UID: \"f5d68385-595f-4d22-b761-e2be6ce315e4\") "
Oct 30 04:50:39.722496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:39.721752    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d68385-595f-4d22-b761-e2be6ce315e4-kube-api-access-95tdk\" (UniqueName: \"kubernetes.io/projected/f5d68385-595f-4d22-b761-e2be6ce315e4-kube-api-access-95tdk\") pod \"f5d68385-595f-4d22-b761-e2be6ce315e4\" (UID: \"f5d68385-595f-4d22-b761-e2be6ce315e4\") "
Oct 30 04:50:39.722496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:39.721881    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-xks-ca\") pod \"f5d68385-595f-4d22-b761-e2be6ce315e4\" (UID: \"f5d68385-595f-4d22-b761-e2be6ce315e4\") "
Oct 30 04:50:39.722887 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:39.722865    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-openshift-ca" pod "f5d68385-595f-4d22-b761-e2be6ce315e4" (UID: "f5d68385-595f-4d22-b761-e2be6ce315e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:50:39.725309 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:39.725286    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-xks-ca" pod "f5d68385-595f-4d22-b761-e2be6ce315e4" (UID: "f5d68385-595f-4d22-b761-e2be6ce315e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:50:39.725401 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:39.725363    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d68385-595f-4d22-b761-e2be6ce315e4-kube-api-access-95tdk" pod "f5d68385-595f-4d22-b761-e2be6ce315e4" (UID: "f5d68385-595f-4d22-b761-e2be6ce315e4"). InnerVolumeSpecName "kube-api-access-95tdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:50:39.823703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:39.823669    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:39.823703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:39.823697    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95tdk\" (UniqueName: \"kubernetes.io/projected/f5d68385-595f-4d22-b761-e2be6ce315e4-kube-api-access-95tdk\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:39.823703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:39.823709    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d68385-595f-4d22-b761-e2be6ce315e4-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:40.454884 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:40.454850    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"
Oct 30 04:50:40.455070 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:40.454849    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk" event={"ID":"f5d68385-595f-4d22-b761-e2be6ce315e4","Type":"ContainerDied","Data":"caab3a206d5e4bd81b9c219f700c2b0d0a7eb8859ed853e8045eae8356c5f127"}
Oct 30 04:50:40.455070 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:40.454983    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caab3a206d5e4bd81b9c219f700c2b0d0a7eb8859ed853e8045eae8356c5f127"
Oct 30 04:50:41.753338 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:41.753306    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:50:43.169382 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:43.169333    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-503804/agc-cred-1b38359d0333: secret "agc-cred-1b38359d0333" not found
Oct 30 04:50:43.169382 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:43.169376    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-503804/agc-cred-e68b135ffabe: secret "agc-cred-e68b135ffabe" not found
Oct 30 04:50:43.169382 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:43.169387    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh: [secret "agc-cred-1b38359d0333" not found, secret "agc-cred-e68b135ffabe" not found]
Oct 30 04:50:43.169972 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:43.169471    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials podName:1c31590c-7aee-47d9-9e1e-1dd95994401a nodeName:}" failed. No retries permitted until 2026-10-30 04:50:59.169450663 +0000 UTC m=+3238.910873755 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-503804-848d98lsh" (UID: "1c31590c-7aee-47d9-9e1e-1dd95994401a") : [secret "agc-cred-1b38359d0333" not found, secret "agc-cred-e68b135ffabe" not found]
Oct 30 04:50:45.515188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:45.515139    5072 generic.go:356] "Generic (PLEG): container finished" podID="04291e68-092a-4c76-bb56-52ce38990484" containerID="cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec" exitCode=0
Oct 30 04:50:45.515656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:45.515213    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" event={"ID":"04291e68-092a-4c76-bb56-52ce38990484","Type":"ContainerDied","Data":"cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec"}
Oct 30 04:50:45.515656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:45.515633    5072 scope.go:122] "RemoveContainer" containerID="cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec"
Oct 30 04:50:45.517744 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:45.517712    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:50:45.517883 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:45.517786    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:50:45.517883 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:45.517828    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:50:45.979741 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:45.979704    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:50:46.525219 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:46.525181    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" event={"ID":"04291e68-092a-4c76-bb56-52ce38990484","Type":"ContainerStarted","Data":"5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f"}
Oct 30 04:50:46.689836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:46.689801    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:50:46.691098 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:46.691009    5072 patch_prober.go:29] interesting pod/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 04:50:46.691098 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:46.691090    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 04:50:47.274764 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:47.274730    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"]
Oct 30 04:50:47.689846 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:47.689810    5072 patch_prober.go:29] interesting pod/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 04:50:47.690496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:47.690462    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 04:50:48.690102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:48.690058    5072 patch_prober.go:29] interesting pod/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 04:50:48.690489 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:48.690121    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 04:50:49.690197 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:49.690158    5072 patch_prober.go:29] interesting pod/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 04:50:49.690698 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:49.690222    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 04:50:50.690651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:50.690613    5072 patch_prober.go:29] interesting pod/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 04:50:50.691070 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:50.690683    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 04:50:51.690767 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:51.690727    5072 patch_prober.go:29] interesting pod/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 04:50:51.691184 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:51.690807    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 04:50:52.276167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.276105    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh" podUID="1c31590c-7aee-47d9-9e1e-1dd95994401a" containerName="payload-processing" containerID="cri-o://2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381" gracePeriod=25
Oct 30 04:50:52.549203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.549175    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:50:52.595328 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.595292    5072 generic.go:356] "Generic (PLEG): container finished" podID="1c31590c-7aee-47d9-9e1e-1dd95994401a" containerID="2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381" exitCode=0
Oct 30 04:50:52.595684 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.595656    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh" event={"ID":"1c31590c-7aee-47d9-9e1e-1dd95994401a","Type":"ContainerDied","Data":"2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381"}
Oct 30 04:50:52.595866 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.595841    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh" event={"ID":"1c31590c-7aee-47d9-9e1e-1dd95994401a","Type":"ContainerDied","Data":"6c764407727454d5dd7a10c050e9615591f5581c750c95ed7f90eb01718478b5"}
Oct 30 04:50:52.595983 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.595881    5072 scope.go:122] "RemoveContainer" containerID="2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381"
Oct 30 04:50:52.596055 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.596040    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"
Oct 30 04:50:52.609471 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.609452    5072 scope.go:122] "RemoveContainer" containerID="2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381"
Oct 30 04:50:52.609840 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:50:52.609819    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381\": container with ID starting with 2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381 not found: ID does not exist" containerID="2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381"
Oct 30 04:50:52.609910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.609848    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381"} err="failed to get container status \"2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381\": rpc error: code = NotFound desc = could not find container \"2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381\": container with ID starting with 2386ba4ae7e36020a6874e67ae192a409385576b46b7fe7cfb9399e87b1f7381 not found: ID does not exist"
Oct 30 04:50:52.695173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.695145    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:50:52.695638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.695551    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:50:52.696253 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.696235    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 04:50:52.719788 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.719762    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-plugins-config-volume\") pod \"1c31590c-7aee-47d9-9e1e-1dd95994401a\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") "
Oct 30 04:50:52.719894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.719830    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-kube-api-access-9sbls\" (UniqueName: \"kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-kube-api-access-9sbls\") pod \"1c31590c-7aee-47d9-9e1e-1dd95994401a\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") "
Oct 30 04:50:52.719957 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.719920    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-routing-overlay-volume\") pod \"1c31590c-7aee-47d9-9e1e-1dd95994401a\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") "
Oct 30 04:50:52.720100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.720062    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials\") pod \"1c31590c-7aee-47d9-9e1e-1dd95994401a\" (UID: \"1c31590c-7aee-47d9-9e1e-1dd95994401a\") "
Oct 30 04:50:52.720233 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.720205    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-plugins-config-volume" pod "1c31590c-7aee-47d9-9e1e-1dd95994401a" (UID: "1c31590c-7aee-47d9-9e1e-1dd95994401a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:50:52.720369 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.720345    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-routing-overlay-volume" pod "1c31590c-7aee-47d9-9e1e-1dd95994401a" (UID: "1c31590c-7aee-47d9-9e1e-1dd95994401a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:50:52.720561 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.720543    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:52.720604 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.720567    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c31590c-7aee-47d9-9e1e-1dd95994401a-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:52.722318 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.722291    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials" pod "1c31590c-7aee-47d9-9e1e-1dd95994401a" (UID: "1c31590c-7aee-47d9-9e1e-1dd95994401a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:50:52.722401 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.722316    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-kube-api-access-9sbls" pod "1c31590c-7aee-47d9-9e1e-1dd95994401a" (UID: "1c31590c-7aee-47d9-9e1e-1dd95994401a"). InnerVolumeSpecName "kube-api-access-9sbls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:50:52.822664 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.822567    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:52.822664 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.822606    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sbls\" (UniqueName: \"kubernetes.io/projected/1c31590c-7aee-47d9-9e1e-1dd95994401a-kube-api-access-9sbls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:50:52.946150 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.946118    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"]
Oct 30 04:50:52.951225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:52.950971    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-848d98lsh"]
Oct 30 04:50:54.409782 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:50:54.409750    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c31590c-7aee-47d9-9e1e-1dd95994401a" path="/var/lib/kubelet/pods/1c31590c-7aee-47d9-9e1e-1dd95994401a/volumes"
Oct 30 04:51:01.643432 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:01.643400    5072 scope.go:122] "RemoveContainer" containerID="607bc75ca21c5fede1e648218a825425cbe68231be559b7b2b9c1100c1aa2a23"
Oct 30 04:51:01.654698 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:01.654670    5072 scope.go:122] "RemoveContainer" containerID="e30a5aa74f30880cdda10a332a5d520010e69eadd366f6d4792c850984b6235f"
Oct 30 04:51:01.667018 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:01.666996    5072 scope.go:122] "RemoveContainer" containerID="af21f07b4e8c5bebab18ddd5bfacdeadda829257d30e25ced3569d99614a5a41"
Oct 30 04:51:01.677577 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:01.677556    5072 scope.go:122] "RemoveContainer" containerID="6db39da4b9da8c2635831a6edcc751dc8576e8c00ecd5cbe2829f8f3520a96ae"
Oct 30 04:51:05.040468 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.040154    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"]
Oct 30 04:51:05.041321 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.041294    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f5d68385-595f-4d22-b761-e2be6ce315e4" containerName="revoke-keys"
Oct 30 04:51:05.041512 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.041494    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d68385-595f-4d22-b761-e2be6ce315e4" containerName="revoke-keys"
Oct 30 04:51:05.041676 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.041666    5072 cpu_manager.go:414] "containerMap: removing container" podUID="1c31590c-7aee-47d9-9e1e-1dd95994401a" containerName="payload-processing"
Oct 30 04:51:05.041754 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.041745    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c31590c-7aee-47d9-9e1e-1dd95994401a" containerName="payload-processing"
Oct 30 04:51:05.042011 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.041991    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c31590c-7aee-47d9-9e1e-1dd95994401a" containerName="payload-processing"
Oct 30 04:51:05.042158 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.042021    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d68385-595f-4d22-b761-e2be6ce315e4" containerName="revoke-keys"
Oct 30 04:51:05.048490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.048464    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:05.067312 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.067275    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:51:05.068054 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.068025    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:51:05.086715 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.086684    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"]
Oct 30 04:51:05.178143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.178071    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1b3aaa1-fxcmp\" (UID: \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:05.178323 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.178182    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb7g5\" (UniqueName: \"kubernetes.io/projected/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-kube-api-access-cb7g5\") pod \"maas-api-revoke-sub-d1b3aaa1-fxcmp\" (UID: \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:05.178323 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.178236    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1b3aaa1-fxcmp\" (UID: \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:05.279833 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.279800    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb7g5\" (UniqueName: \"kubernetes.io/projected/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-kube-api-access-cb7g5\") pod \"maas-api-revoke-sub-d1b3aaa1-fxcmp\" (UID: \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:05.280014 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.279865    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1b3aaa1-fxcmp\" (UID: \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:05.280014 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.279981    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1b3aaa1-fxcmp\" (UID: \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:05.280623 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.280590    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1b3aaa1-fxcmp\" (UID: \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:05.282754 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.282727    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1b3aaa1-fxcmp\" (UID: \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:05.303913 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.303853    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb7g5\" (UniqueName: \"kubernetes.io/projected/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-kube-api-access-cb7g5\") pod \"maas-api-revoke-sub-d1b3aaa1-fxcmp\" (UID: \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:05.405605 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.405581    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:05.714766 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:51:05.714727    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc076262_cc82_4d9a_a09e_f3f789ca6e6c.slice/crio-b2794adcc890450e2638cb8cc778bab57fa6b7c1a9dda0546221d344752eccc2 WatchSource:0}: Error finding container b2794adcc890450e2638cb8cc778bab57fa6b7c1a9dda0546221d344752eccc2: Status 404 returned error can't find the container with id b2794adcc890450e2638cb8cc778bab57fa6b7c1a9dda0546221d344752eccc2
Oct 30 04:51:05.715403 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:05.715362    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"]
Oct 30 04:51:06.719148 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:06.719101    5072 generic.go:356] "Generic (PLEG): container finished" podID="fc076262-cc82-4d9a-a09e-f3f789ca6e6c" containerID="9ae6a33c453849700518984e9b96b34e4abaeedc3abcc17699b5cbbd77d6e97d" exitCode=0
Oct 30 04:51:06.719148 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:06.719140    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp" event={"ID":"fc076262-cc82-4d9a-a09e-f3f789ca6e6c","Type":"ContainerDied","Data":"9ae6a33c453849700518984e9b96b34e4abaeedc3abcc17699b5cbbd77d6e97d"}
Oct 30 04:51:06.719634 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:06.719202    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp" event={"ID":"fc076262-cc82-4d9a-a09e-f3f789ca6e6c","Type":"ContainerStarted","Data":"b2794adcc890450e2638cb8cc778bab57fa6b7c1a9dda0546221d344752eccc2"}
Oct 30 04:51:07.897622 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:07.890840    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:08.017228 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:08.017144    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-openshift-ca\") pod \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\" (UID: \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\") "
Oct 30 04:51:08.017389 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:08.017233    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-xks-ca\") pod \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\" (UID: \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\") "
Oct 30 04:51:08.017389 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:08.017321    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-kube-api-access-cb7g5\" (UniqueName: \"kubernetes.io/projected/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-kube-api-access-cb7g5\") pod \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\" (UID: \"fc076262-cc82-4d9a-a09e-f3f789ca6e6c\") "
Oct 30 04:51:08.017552 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:08.017514    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-openshift-ca" pod "fc076262-cc82-4d9a-a09e-f3f789ca6e6c" (UID: "fc076262-cc82-4d9a-a09e-f3f789ca6e6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:51:08.017752 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:08.017732    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:08.019634 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:08.019603    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-xks-ca" pod "fc076262-cc82-4d9a-a09e-f3f789ca6e6c" (UID: "fc076262-cc82-4d9a-a09e-f3f789ca6e6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:51:08.019747 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:08.019674    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-kube-api-access-cb7g5" pod "fc076262-cc82-4d9a-a09e-f3f789ca6e6c" (UID: "fc076262-cc82-4d9a-a09e-f3f789ca6e6c"). InnerVolumeSpecName "kube-api-access-cb7g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:51:08.118504 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:08.118474    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:08.118504 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:08.118497    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb7g5\" (UniqueName: \"kubernetes.io/projected/fc076262-cc82-4d9a-a09e-f3f789ca6e6c-kube-api-access-cb7g5\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:08.734831 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:08.734789    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp" event={"ID":"fc076262-cc82-4d9a-a09e-f3f789ca6e6c","Type":"ContainerDied","Data":"b2794adcc890450e2638cb8cc778bab57fa6b7c1a9dda0546221d344752eccc2"}
Oct 30 04:51:08.734831 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:08.734824    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"
Oct 30 04:51:08.735045 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:08.734851    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2794adcc890450e2638cb8cc778bab57fa6b7c1a9dda0546221d344752eccc2"
Oct 30 04:51:11.753632 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:51:11.753596    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:51:19.039347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:19.039310    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"]
Oct 30 04:51:19.051460 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:19.051432    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebcf114-hp4rg"]
Oct 30 04:51:20.410250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:20.410220    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394b0d01-4520-4eb6-8490-719818e3ad43" path="/var/lib/kubelet/pods/394b0d01-4520-4eb6-8490-719818e3ad43/volumes"
Oct 30 04:51:30.031827 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:30.031791    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"]
Oct 30 04:51:30.043995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:30.043966    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b963057-vltsh"]
Oct 30 04:51:30.410667 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:30.410593    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6403d038-c61d-4887-affe-adc433266f9a" path="/var/lib/kubelet/pods/6403d038-c61d-4887-affe-adc433266f9a/volumes"
Oct 30 04:51:32.532181 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.532149    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"]
Oct 30 04:51:32.533117 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.533093    5072 cpu_manager.go:414] "containerMap: removing container" podUID="fc076262-cc82-4d9a-a09e-f3f789ca6e6c" containerName="revoke-keys"
Oct 30 04:51:32.533117 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.533115    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc076262-cc82-4d9a-a09e-f3f789ca6e6c" containerName="revoke-keys"
Oct 30 04:51:32.533334 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.533320    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc076262-cc82-4d9a-a09e-f3f789ca6e6c" containerName="revoke-keys"
Oct 30 04:51:32.537424 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.537401    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:32.554907 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.554882    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"]
Oct 30 04:51:32.563394 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.563374    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:51:32.563629 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.563593    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:51:32.615916 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.615884    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-692dddcb-d64wg\" (UID: \"c6f6c0f0-7776-488c-ba54-756479ac002b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:32.616041 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.615970    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkh72\" (UniqueName: \"kubernetes.io/projected/c6f6c0f0-7776-488c-ba54-756479ac002b-kube-api-access-wkh72\") pod \"maas-api-revoke-sub-692dddcb-d64wg\" (UID: \"c6f6c0f0-7776-488c-ba54-756479ac002b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:32.616041 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.616027    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-692dddcb-d64wg\" (UID: \"c6f6c0f0-7776-488c-ba54-756479ac002b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:32.718013 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.717973    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkh72\" (UniqueName: \"kubernetes.io/projected/c6f6c0f0-7776-488c-ba54-756479ac002b-kube-api-access-wkh72\") pod \"maas-api-revoke-sub-692dddcb-d64wg\" (UID: \"c6f6c0f0-7776-488c-ba54-756479ac002b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:32.718187 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.718070    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-692dddcb-d64wg\" (UID: \"c6f6c0f0-7776-488c-ba54-756479ac002b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:32.718357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.718336    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-692dddcb-d64wg\" (UID: \"c6f6c0f0-7776-488c-ba54-756479ac002b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:32.719216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.719192    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-692dddcb-d64wg\" (UID: \"c6f6c0f0-7776-488c-ba54-756479ac002b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:32.722674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.722640    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-692dddcb-d64wg\" (UID: \"c6f6c0f0-7776-488c-ba54-756479ac002b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:32.745574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.745548    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkh72\" (UniqueName: \"kubernetes.io/projected/c6f6c0f0-7776-488c-ba54-756479ac002b-kube-api-access-wkh72\") pod \"maas-api-revoke-sub-692dddcb-d64wg\" (UID: \"c6f6c0f0-7776-488c-ba54-756479ac002b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:32.853551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.852805    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:32.859311 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.859284    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"]
Oct 30 04:51:32.864095 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.864058    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:32.884098 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.884049    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"]
Oct 30 04:51:32.921846 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.921810    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9e45d13-ghvxq\" (UID: \"91788bca-5e08-4ec9-8011-2162a3db043e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:32.922021 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.921861    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9e45d13-ghvxq\" (UID: \"91788bca-5e08-4ec9-8011-2162a3db043e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:32.922021 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:32.921913    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkz8m\" (UniqueName: \"kubernetes.io/projected/91788bca-5e08-4ec9-8011-2162a3db043e-kube-api-access-dkz8m\") pod \"maas-api-revoke-sub-d9e45d13-ghvxq\" (UID: \"91788bca-5e08-4ec9-8011-2162a3db043e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:33.023835 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:33.023796    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9e45d13-ghvxq\" (UID: \"91788bca-5e08-4ec9-8011-2162a3db043e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:33.024000 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:33.023851    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9e45d13-ghvxq\" (UID: \"91788bca-5e08-4ec9-8011-2162a3db043e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:33.024000 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:33.023905    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkz8m\" (UniqueName: \"kubernetes.io/projected/91788bca-5e08-4ec9-8011-2162a3db043e-kube-api-access-dkz8m\") pod \"maas-api-revoke-sub-d9e45d13-ghvxq\" (UID: \"91788bca-5e08-4ec9-8011-2162a3db043e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:33.024838 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:33.024809    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9e45d13-ghvxq\" (UID: \"91788bca-5e08-4ec9-8011-2162a3db043e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:33.026851 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:33.026822    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9e45d13-ghvxq\" (UID: \"91788bca-5e08-4ec9-8011-2162a3db043e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:33.048443 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:33.048413    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkz8m\" (UniqueName: \"kubernetes.io/projected/91788bca-5e08-4ec9-8011-2162a3db043e-kube-api-access-dkz8m\") pod \"maas-api-revoke-sub-d9e45d13-ghvxq\" (UID: \"91788bca-5e08-4ec9-8011-2162a3db043e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:33.230106 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:33.225876    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"]
Oct 30 04:51:33.230262 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:33.230192    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:33.552839 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:51:33.552805    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91788bca_5e08_4ec9_8011_2162a3db043e.slice/crio-e66bbf88be5bfc971860466ceccdd29eb61c09de88f91e8505922cd479edb1d2 WatchSource:0}: Error finding container e66bbf88be5bfc971860466ceccdd29eb61c09de88f91e8505922cd479edb1d2: Status 404 returned error can't find the container with id e66bbf88be5bfc971860466ceccdd29eb61c09de88f91e8505922cd479edb1d2
Oct 30 04:51:33.553372 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:33.553009    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"]
Oct 30 04:51:33.937918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:33.937882    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq" event={"ID":"91788bca-5e08-4ec9-8011-2162a3db043e","Type":"ContainerStarted","Data":"e66bbf88be5bfc971860466ceccdd29eb61c09de88f91e8505922cd479edb1d2"}
Oct 30 04:51:33.939678 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:33.939650    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg" event={"ID":"c6f6c0f0-7776-488c-ba54-756479ac002b","Type":"ContainerStarted","Data":"53d96b3b9476e7b21dbcbb7e5ac13b7b9c420c782a64d5240e74f8e9bb876452"}
Oct 30 04:51:33.939819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:33.939691    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg" event={"ID":"c6f6c0f0-7776-488c-ba54-756479ac002b","Type":"ContainerStarted","Data":"dd8e2ada7d3868b7963a80d807f3b20685c88a6463739c1c18de48891cbbcd14"}
Oct 30 04:51:34.000788 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:34.000731    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg" podStartSLOduration=2.000716103 podStartE2EDuration="2.000716103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:33.999692667 +0000 UTC m=+3273.741115765" watchObservedRunningTime="2026-10-30 04:51:34.000716103 +0000 UTC m=+3273.742139200"
Oct 30 04:51:34.948667 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:34.948634    5072 generic.go:356] "Generic (PLEG): container finished" podID="91788bca-5e08-4ec9-8011-2162a3db043e" containerID="7c454bfdfb60f1aad85a4baa6ed09e890a17d5c7fb6e95d6c934832021b41510" exitCode=0
Oct 30 04:51:34.949160 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:34.948724    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq" event={"ID":"91788bca-5e08-4ec9-8011-2162a3db043e","Type":"ContainerDied","Data":"7c454bfdfb60f1aad85a4baa6ed09e890a17d5c7fb6e95d6c934832021b41510"}
Oct 30 04:51:34.950330 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:34.950300    5072 generic.go:356] "Generic (PLEG): container finished" podID="c6f6c0f0-7776-488c-ba54-756479ac002b" containerID="53d96b3b9476e7b21dbcbb7e5ac13b7b9c420c782a64d5240e74f8e9bb876452" exitCode=0
Oct 30 04:51:34.950454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:34.950369    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg" event={"ID":"c6f6c0f0-7776-488c-ba54-756479ac002b","Type":"ContainerDied","Data":"53d96b3b9476e7b21dbcbb7e5ac13b7b9c420c782a64d5240e74f8e9bb876452"}
Oct 30 04:51:36.160899 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.160872    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:36.164599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.164576    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:36.262194 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.262117    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-openshift-ca\") pod \"c6f6c0f0-7776-488c-ba54-756479ac002b\" (UID: \"c6f6c0f0-7776-488c-ba54-756479ac002b\") "
Oct 30 04:51:36.262194 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.262169    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91788bca-5e08-4ec9-8011-2162a3db043e-kube-api-access-dkz8m\" (UniqueName: \"kubernetes.io/projected/91788bca-5e08-4ec9-8011-2162a3db043e-kube-api-access-dkz8m\") pod \"91788bca-5e08-4ec9-8011-2162a3db043e\" (UID: \"91788bca-5e08-4ec9-8011-2162a3db043e\") "
Oct 30 04:51:36.262439 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.262197    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-xks-ca\") pod \"c6f6c0f0-7776-488c-ba54-756479ac002b\" (UID: \"c6f6c0f0-7776-488c-ba54-756479ac002b\") "
Oct 30 04:51:36.262439 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.262244    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f6c0f0-7776-488c-ba54-756479ac002b-kube-api-access-wkh72\" (UniqueName: \"kubernetes.io/projected/c6f6c0f0-7776-488c-ba54-756479ac002b-kube-api-access-wkh72\") pod \"c6f6c0f0-7776-488c-ba54-756479ac002b\" (UID: \"c6f6c0f0-7776-488c-ba54-756479ac002b\") "
Oct 30 04:51:36.262439 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.262325    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-openshift-ca\") pod \"91788bca-5e08-4ec9-8011-2162a3db043e\" (UID: \"91788bca-5e08-4ec9-8011-2162a3db043e\") "
Oct 30 04:51:36.262639 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.262500    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-xks-ca\") pod \"91788bca-5e08-4ec9-8011-2162a3db043e\" (UID: \"91788bca-5e08-4ec9-8011-2162a3db043e\") "
Oct 30 04:51:36.262639 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.262547    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-openshift-ca" pod "c6f6c0f0-7776-488c-ba54-756479ac002b" (UID: "c6f6c0f0-7776-488c-ba54-756479ac002b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:51:36.262996 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.262970    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-openshift-ca" pod "91788bca-5e08-4ec9-8011-2162a3db043e" (UID: "91788bca-5e08-4ec9-8011-2162a3db043e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:51:36.263166 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.262990    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:36.265823 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.265799    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91788bca-5e08-4ec9-8011-2162a3db043e-kube-api-access-dkz8m" pod "91788bca-5e08-4ec9-8011-2162a3db043e" (UID: "91788bca-5e08-4ec9-8011-2162a3db043e"). InnerVolumeSpecName "kube-api-access-dkz8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:51:36.265933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.265827    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f6c0f0-7776-488c-ba54-756479ac002b-kube-api-access-wkh72" pod "c6f6c0f0-7776-488c-ba54-756479ac002b" (UID: "c6f6c0f0-7776-488c-ba54-756479ac002b"). InnerVolumeSpecName "kube-api-access-wkh72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:51:36.266001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.265957    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-xks-ca" pod "c6f6c0f0-7776-488c-ba54-756479ac002b" (UID: "c6f6c0f0-7776-488c-ba54-756479ac002b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:51:36.267582 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.267552    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-xks-ca" pod "91788bca-5e08-4ec9-8011-2162a3db043e" (UID: "91788bca-5e08-4ec9-8011-2162a3db043e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:51:36.364114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.364056    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:36.364114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.364100    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkz8m\" (UniqueName: \"kubernetes.io/projected/91788bca-5e08-4ec9-8011-2162a3db043e-kube-api-access-dkz8m\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:36.364114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.364111    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f6c0f0-7776-488c-ba54-756479ac002b-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:36.364345 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.364128    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkh72\" (UniqueName: \"kubernetes.io/projected/c6f6c0f0-7776-488c-ba54-756479ac002b-kube-api-access-wkh72\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:36.364345 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.364138    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91788bca-5e08-4ec9-8011-2162a3db043e-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:36.967788 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.967761    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"
Oct 30 04:51:36.967788 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.967766    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq" event={"ID":"91788bca-5e08-4ec9-8011-2162a3db043e","Type":"ContainerDied","Data":"e66bbf88be5bfc971860466ceccdd29eb61c09de88f91e8505922cd479edb1d2"}
Oct 30 04:51:36.968058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.967819    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e66bbf88be5bfc971860466ceccdd29eb61c09de88f91e8505922cd479edb1d2"
Oct 30 04:51:36.969523 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.969501    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"
Oct 30 04:51:36.969659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.969501    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg" event={"ID":"c6f6c0f0-7776-488c-ba54-756479ac002b","Type":"ContainerDied","Data":"dd8e2ada7d3868b7963a80d807f3b20685c88a6463739c1c18de48891cbbcd14"}
Oct 30 04:51:36.969659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:36.969634    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd8e2ada7d3868b7963a80d807f3b20685c88a6463739c1c18de48891cbbcd14"
Oct 30 04:51:41.752034 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:51:41.751948    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:51:43.642459 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.642427    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"]
Oct 30 04:51:43.643371 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.643348    5072 cpu_manager.go:414] "containerMap: removing container" podUID="c6f6c0f0-7776-488c-ba54-756479ac002b" containerName="revoke-keys"
Oct 30 04:51:43.643371 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.643371    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f6c0f0-7776-488c-ba54-756479ac002b" containerName="revoke-keys"
Oct 30 04:51:43.643555 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.643402    5072 cpu_manager.go:414] "containerMap: removing container" podUID="91788bca-5e08-4ec9-8011-2162a3db043e" containerName="revoke-keys"
Oct 30 04:51:43.643555 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.643409    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91788bca-5e08-4ec9-8011-2162a3db043e" containerName="revoke-keys"
Oct 30 04:51:43.643667 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.643627    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="91788bca-5e08-4ec9-8011-2162a3db043e" containerName="revoke-keys"
Oct 30 04:51:43.643667 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.643656    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f6c0f0-7776-488c-ba54-756479ac002b" containerName="revoke-keys"
Oct 30 04:51:43.648559 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.648537    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:43.663562 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.663534    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"]
Oct 30 04:51:43.672336 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.672312    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:51:43.672457 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.672401    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:51:43.750920 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.750892    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ac36228-bf88m\" (UID: \"149f3d6b-e599-41ed-be75-c3311ac3c061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:43.750920 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.750920    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfr7c\" (UniqueName: \"kubernetes.io/projected/149f3d6b-e599-41ed-be75-c3311ac3c061-kube-api-access-pfr7c\") pod \"maas-api-revoke-sub-0ac36228-bf88m\" (UID: \"149f3d6b-e599-41ed-be75-c3311ac3c061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:43.751166 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.751036    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ac36228-bf88m\" (UID: \"149f3d6b-e599-41ed-be75-c3311ac3c061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:43.837205 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.837179    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"]
Oct 30 04:51:43.841752 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.841734    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:43.852820 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.852797    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ac36228-bf88m\" (UID: \"149f3d6b-e599-41ed-be75-c3311ac3c061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:43.852910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.852825    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfr7c\" (UniqueName: \"kubernetes.io/projected/149f3d6b-e599-41ed-be75-c3311ac3c061-kube-api-access-pfr7c\") pod \"maas-api-revoke-sub-0ac36228-bf88m\" (UID: \"149f3d6b-e599-41ed-be75-c3311ac3c061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:43.852910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.852882    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ac36228-bf88m\" (UID: \"149f3d6b-e599-41ed-be75-c3311ac3c061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:43.853421 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.853399    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ac36228-bf88m\" (UID: \"149f3d6b-e599-41ed-be75-c3311ac3c061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:43.855345 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.855324    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ac36228-bf88m\" (UID: \"149f3d6b-e599-41ed-be75-c3311ac3c061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:43.867028 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.867003    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"]
Oct 30 04:51:43.877597 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.877575    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfr7c\" (UniqueName: \"kubernetes.io/projected/149f3d6b-e599-41ed-be75-c3311ac3c061-kube-api-access-pfr7c\") pod \"maas-api-revoke-sub-0ac36228-bf88m\" (UID: \"149f3d6b-e599-41ed-be75-c3311ac3c061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:43.954026 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.953984    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcae8d78-pzjxq\" (UID: \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:43.954223 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.954138    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcae8d78-pzjxq\" (UID: \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:43.954294 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.954255    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvff\" (UniqueName: \"kubernetes.io/projected/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-kube-api-access-ccvff\") pod \"maas-api-revoke-sub-fcae8d78-pzjxq\" (UID: \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:43.963742 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:43.963721    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:44.055392 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:44.055357    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcae8d78-pzjxq\" (UID: \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:44.055580 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:44.055560    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcae8d78-pzjxq\" (UID: \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:44.055651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:44.055638    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvff\" (UniqueName: \"kubernetes.io/projected/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-kube-api-access-ccvff\") pod \"maas-api-revoke-sub-fcae8d78-pzjxq\" (UID: \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:44.056727 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:44.056702    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcae8d78-pzjxq\" (UID: \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:44.058686 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:44.058641    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcae8d78-pzjxq\" (UID: \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:44.079564 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:44.079542    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvff\" (UniqueName: \"kubernetes.io/projected/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-kube-api-access-ccvff\") pod \"maas-api-revoke-sub-fcae8d78-pzjxq\" (UID: \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:44.156275 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:44.156247    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:44.277206 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:44.277177    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"]
Oct 30 04:51:44.277377 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:51:44.277356    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149f3d6b_e599_41ed_be75_c3311ac3c061.slice/crio-f729029c82c8869d902575ce8a2d2e91c71cd2f3529673e00df51e841c49a6bb WatchSource:0}: Error finding container f729029c82c8869d902575ce8a2d2e91c71cd2f3529673e00df51e841c49a6bb: Status 404 returned error can't find the container with id f729029c82c8869d902575ce8a2d2e91c71cd2f3529673e00df51e841c49a6bb
Oct 30 04:51:44.461674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:44.461615    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"]
Oct 30 04:51:44.472554 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:51:44.472515    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568b2434_830f_4e8f_9f96_42c8cf3f6ffa.slice/crio-87a718c10d7d85d833b7cc52ddb74981108f57d418c63ddabbc02132f3f4c47b WatchSource:0}: Error finding container 87a718c10d7d85d833b7cc52ddb74981108f57d418c63ddabbc02132f3f4c47b: Status 404 returned error can't find the container with id 87a718c10d7d85d833b7cc52ddb74981108f57d418c63ddabbc02132f3f4c47b
Oct 30 04:51:45.041135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:45.040916    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m" event={"ID":"149f3d6b-e599-41ed-be75-c3311ac3c061","Type":"ContainerStarted","Data":"334872e6a23d73d1557617bde9973355c0ff3ce579d42868fa0f2d6193f37082"}
Oct 30 04:51:45.041135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:45.041056    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m" event={"ID":"149f3d6b-e599-41ed-be75-c3311ac3c061","Type":"ContainerStarted","Data":"f729029c82c8869d902575ce8a2d2e91c71cd2f3529673e00df51e841c49a6bb"}
Oct 30 04:51:45.044148 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:45.044115    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq" event={"ID":"568b2434-830f-4e8f-9f96-42c8cf3f6ffa","Type":"ContainerStarted","Data":"caab2887e92f2678aa9db511df7277282946716a7a7b3e02f4ab88430655c44c"}
Oct 30 04:51:45.044503 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:45.044198    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq" event={"ID":"568b2434-830f-4e8f-9f96-42c8cf3f6ffa","Type":"ContainerStarted","Data":"87a718c10d7d85d833b7cc52ddb74981108f57d418c63ddabbc02132f3f4c47b"}
Oct 30 04:51:45.085281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:45.085231    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m" podStartSLOduration=2.085217745 podStartE2EDuration="2.085217745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:45.082793976 +0000 UTC m=+3284.824217072" watchObservedRunningTime="2026-10-30 04:51:45.085217745 +0000 UTC m=+3284.826640841"
Oct 30 04:51:45.127181 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:45.127121    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq" podStartSLOduration=2.127103783 podStartE2EDuration="2.127103783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:45.123995624 +0000 UTC m=+3284.865418722" watchObservedRunningTime="2026-10-30 04:51:45.127103783 +0000 UTC m=+3284.868526882"
Oct 30 04:51:46.053229 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:46.053195    5072 generic.go:356] "Generic (PLEG): container finished" podID="149f3d6b-e599-41ed-be75-c3311ac3c061" containerID="334872e6a23d73d1557617bde9973355c0ff3ce579d42868fa0f2d6193f37082" exitCode=0
Oct 30 04:51:46.053674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:46.053254    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m" event={"ID":"149f3d6b-e599-41ed-be75-c3311ac3c061","Type":"ContainerDied","Data":"334872e6a23d73d1557617bde9973355c0ff3ce579d42868fa0f2d6193f37082"}
Oct 30 04:51:46.054977 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:46.054951    5072 generic.go:356] "Generic (PLEG): container finished" podID="568b2434-830f-4e8f-9f96-42c8cf3f6ffa" containerID="caab2887e92f2678aa9db511df7277282946716a7a7b3e02f4ab88430655c44c" exitCode=0
Oct 30 04:51:46.055118 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:46.055032    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq" event={"ID":"568b2434-830f-4e8f-9f96-42c8cf3f6ffa","Type":"ContainerDied","Data":"caab2887e92f2678aa9db511df7277282946716a7a7b3e02f4ab88430655c44c"}
Oct 30 04:51:47.230325 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.230291    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:47.264790 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.264764    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:47.394522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.394439    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-openshift-ca\") pod \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\" (UID: \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\") "
Oct 30 04:51:47.394678 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.394530    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/149f3d6b-e599-41ed-be75-c3311ac3c061-kube-api-access-pfr7c\" (UniqueName: \"kubernetes.io/projected/149f3d6b-e599-41ed-be75-c3311ac3c061-kube-api-access-pfr7c\") pod \"149f3d6b-e599-41ed-be75-c3311ac3c061\" (UID: \"149f3d6b-e599-41ed-be75-c3311ac3c061\") "
Oct 30 04:51:47.394678 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.394636    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-xks-ca\") pod \"149f3d6b-e599-41ed-be75-c3311ac3c061\" (UID: \"149f3d6b-e599-41ed-be75-c3311ac3c061\") "
Oct 30 04:51:47.394678 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.394671    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-kube-api-access-ccvff\" (UniqueName: \"kubernetes.io/projected/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-kube-api-access-ccvff\") pod \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\" (UID: \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\") "
Oct 30 04:51:47.394859 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.394719    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-xks-ca\") pod \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\" (UID: \"568b2434-830f-4e8f-9f96-42c8cf3f6ffa\") "
Oct 30 04:51:47.394929 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.394860    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-openshift-ca\") pod \"149f3d6b-e599-41ed-be75-c3311ac3c061\" (UID: \"149f3d6b-e599-41ed-be75-c3311ac3c061\") "
Oct 30 04:51:47.395129 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.394866    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-openshift-ca" pod "568b2434-830f-4e8f-9f96-42c8cf3f6ffa" (UID: "568b2434-830f-4e8f-9f96-42c8cf3f6ffa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:51:47.395387 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.395261    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-openshift-ca" pod "149f3d6b-e599-41ed-be75-c3311ac3c061" (UID: "149f3d6b-e599-41ed-be75-c3311ac3c061"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:51:47.395722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.395668    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:47.395800 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.395730    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:47.397035 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.397002    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-xks-ca" pod "568b2434-830f-4e8f-9f96-42c8cf3f6ffa" (UID: "568b2434-830f-4e8f-9f96-42c8cf3f6ffa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:51:47.397163 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.397040    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149f3d6b-e599-41ed-be75-c3311ac3c061-kube-api-access-pfr7c" pod "149f3d6b-e599-41ed-be75-c3311ac3c061" (UID: "149f3d6b-e599-41ed-be75-c3311ac3c061"). InnerVolumeSpecName "kube-api-access-pfr7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:51:47.397211 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.397170    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-xks-ca" pod "149f3d6b-e599-41ed-be75-c3311ac3c061" (UID: "149f3d6b-e599-41ed-be75-c3311ac3c061"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:51:47.397395 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.397373    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-kube-api-access-ccvff" pod "568b2434-830f-4e8f-9f96-42c8cf3f6ffa" (UID: "568b2434-830f-4e8f-9f96-42c8cf3f6ffa"). InnerVolumeSpecName "kube-api-access-ccvff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:51:47.496748 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.496715    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfr7c\" (UniqueName: \"kubernetes.io/projected/149f3d6b-e599-41ed-be75-c3311ac3c061-kube-api-access-pfr7c\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:47.496885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.496835    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149f3d6b-e599-41ed-be75-c3311ac3c061-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:47.496885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.496850    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccvff\" (UniqueName: \"kubernetes.io/projected/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-kube-api-access-ccvff\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:47.496885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:47.496859    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568b2434-830f-4e8f-9f96-42c8cf3f6ffa-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:51:48.070907 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:48.070877    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"
Oct 30 04:51:48.071058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:48.070908    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq" event={"ID":"568b2434-830f-4e8f-9f96-42c8cf3f6ffa","Type":"ContainerDied","Data":"87a718c10d7d85d833b7cc52ddb74981108f57d418c63ddabbc02132f3f4c47b"}
Oct 30 04:51:48.071058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:48.070955    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87a718c10d7d85d833b7cc52ddb74981108f57d418c63ddabbc02132f3f4c47b"
Oct 30 04:51:48.072634 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:48.072607    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m" event={"ID":"149f3d6b-e599-41ed-be75-c3311ac3c061","Type":"ContainerDied","Data":"f729029c82c8869d902575ce8a2d2e91c71cd2f3529673e00df51e841c49a6bb"}
Oct 30 04:51:48.072749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:48.072648    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f729029c82c8869d902575ce8a2d2e91c71cd2f3529673e00df51e841c49a6bb"
Oct 30 04:51:48.072749 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:48.072619    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"
Oct 30 04:51:56.080430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.080387    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"]
Oct 30 04:51:56.081642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.081611    5072 cpu_manager.go:414] "containerMap: removing container" podUID="149f3d6b-e599-41ed-be75-c3311ac3c061" containerName="revoke-keys"
Oct 30 04:51:56.081642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.081638    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149f3d6b-e599-41ed-be75-c3311ac3c061" containerName="revoke-keys"
Oct 30 04:51:56.081826 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.081696    5072 cpu_manager.go:414] "containerMap: removing container" podUID="568b2434-830f-4e8f-9f96-42c8cf3f6ffa" containerName="revoke-keys"
Oct 30 04:51:56.081826 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.081705    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568b2434-830f-4e8f-9f96-42c8cf3f6ffa" containerName="revoke-keys"
Oct 30 04:51:56.081936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.081892    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="149f3d6b-e599-41ed-be75-c3311ac3c061" containerName="revoke-keys"
Oct 30 04:51:56.081936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.081915    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="568b2434-830f-4e8f-9f96-42c8cf3f6ffa" containerName="revoke-keys"
Oct 30 04:51:56.087343 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.087320    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.097892 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.097866    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.098244 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.098221    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.098533 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.098511    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.098730 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.098709    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlxx\" (UniqueName: \"kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-kube-api-access-ndlxx\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.108923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.108899    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"]
Oct 30 04:51:56.204897 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.204860    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.205143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.204974    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.205143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.205002    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlxx\" (UniqueName: \"kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-kube-api-access-ndlxx\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.205270 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.205138    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.206298 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.206270    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.207888 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.207857    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.209257 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.209239    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.238339 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.237358    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlxx\" (UniqueName: \"kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-kube-api-access-ndlxx\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.414655 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.414561    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:56.678503 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.677837    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"]
Oct 30 04:51:56.751652 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:51:56.751054    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93374e47_d35f_4333_b466_d8c2ece570e0.slice/crio-e2ac75650118aee39f7f1530d2b77b10e89f4b77e1f927c37fbd038f67f393a6 WatchSource:0}: Error finding container e2ac75650118aee39f7f1530d2b77b10e89f4b77e1f927c37fbd038f67f393a6: Status 404 returned error can't find the container with id e2ac75650118aee39f7f1530d2b77b10e89f4b77e1f927c37fbd038f67f393a6
Oct 30 04:51:56.752527 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:56.752502    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"]
Oct 30 04:51:57.172959 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:57.172922    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87" event={"ID":"93374e47-d35f-4333-b466-d8c2ece570e0","Type":"ContainerStarted","Data":"9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b"}
Oct 30 04:51:57.173462 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:57.172991    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87" event={"ID":"93374e47-d35f-4333-b466-d8c2ece570e0","Type":"ContainerStarted","Data":"e2ac75650118aee39f7f1530d2b77b10e89f4b77e1f927c37fbd038f67f393a6"}
Oct 30 04:51:57.173462 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:57.173193    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:51:57.240169 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:57.240097    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87" podStartSLOduration=1.240062358 podStartE2EDuration="1.240062358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:57.235351428 +0000 UTC m=+3296.976774526" watchObservedRunningTime="2026-10-30 04:51:57.240062358 +0000 UTC m=+3296.981485456"
Oct 30 04:51:57.242678 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:51:57.241842    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-ed8266/agc-cred-16d391637b1c: secret "agc-cred-16d391637b1c" not found
Oct 30 04:51:57.242678 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:51:57.241870    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87: secret "agc-cred-16d391637b1c" not found
Oct 30 04:51:57.242678 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:51:57.241924    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials podName:93374e47-d35f-4333-b466-d8c2ece570e0 nodeName:}" failed. No retries permitted until 2026-10-30 04:51:57.741905689 +0000 UTC m=+3297.483328768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87" (UID: "93374e47-d35f-4333-b466-d8c2ece570e0") : secret "agc-cred-16d391637b1c" not found
Oct 30 04:51:57.751946 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:51:57.751913    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-ed8266/agc-cred-16d391637b1c: secret "agc-cred-16d391637b1c" not found
Oct 30 04:51:57.751946 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:51:57.751937    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87: secret "agc-cred-16d391637b1c" not found
Oct 30 04:51:57.752196 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:51:57.751989    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials podName:93374e47-d35f-4333-b466-d8c2ece570e0 nodeName:}" failed. No retries permitted until 2026-10-30 04:51:58.751974565 +0000 UTC m=+3298.493397672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87" (UID: "93374e47-d35f-4333-b466-d8c2ece570e0") : secret "agc-cred-16d391637b1c" not found
Oct 30 04:51:58.247782 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.247747    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"]
Oct 30 04:51:58.252249 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.252228    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:51:58.272157 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.272054    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"]
Oct 30 04:51:58.280058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.279049    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:51:58.280058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.279102    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:51:58.348167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.348135    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"]
Oct 30 04:51:58.351858 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.351834    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"]
Oct 30 04:51:58.358339 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.358308    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4859cfc6-sxpj8\" (UID: \"2907e444-12e2-47cd-b6d8-b5b4183ec275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:51:58.358696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.358661    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwd7k\" (UniqueName: \"kubernetes.io/projected/2907e444-12e2-47cd-b6d8-b5b4183ec275-kube-api-access-dwd7k\") pod \"maas-api-revoke-sub-4859cfc6-sxpj8\" (UID: \"2907e444-12e2-47cd-b6d8-b5b4183ec275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:51:58.358803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.358702    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4859cfc6-sxpj8\" (UID: \"2907e444-12e2-47cd-b6d8-b5b4183ec275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:51:58.460330 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.460295    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd7k\" (UniqueName: \"kubernetes.io/projected/2907e444-12e2-47cd-b6d8-b5b4183ec275-kube-api-access-dwd7k\") pod \"maas-api-revoke-sub-4859cfc6-sxpj8\" (UID: \"2907e444-12e2-47cd-b6d8-b5b4183ec275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:51:58.460526 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.460335    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4859cfc6-sxpj8\" (UID: \"2907e444-12e2-47cd-b6d8-b5b4183ec275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:51:58.460526 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.460357    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4859cfc6-sxpj8\" (UID: \"2907e444-12e2-47cd-b6d8-b5b4183ec275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:51:58.461119 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.461071    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4859cfc6-sxpj8\" (UID: \"2907e444-12e2-47cd-b6d8-b5b4183ec275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:51:58.463195 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.463164    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4859cfc6-sxpj8\" (UID: \"2907e444-12e2-47cd-b6d8-b5b4183ec275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:51:58.485976 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.485946    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwd7k\" (UniqueName: \"kubernetes.io/projected/2907e444-12e2-47cd-b6d8-b5b4183ec275-kube-api-access-dwd7k\") pod \"maas-api-revoke-sub-4859cfc6-sxpj8\" (UID: \"2907e444-12e2-47cd-b6d8-b5b4183ec275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:51:58.568089 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.568011    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:51:58.766185 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:51:58.766149    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-ed8266/agc-cred-16d391637b1c: secret "agc-cred-16d391637b1c" not found
Oct 30 04:51:58.766185 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:51:58.766179    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87: secret "agc-cred-16d391637b1c" not found
Oct 30 04:51:58.766407 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:51:58.766240    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials podName:93374e47-d35f-4333-b466-d8c2ece570e0 nodeName:}" failed. No retries permitted until 2026-10-30 04:52:00.766221855 +0000 UTC m=+3300.507644938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87" (UID: "93374e47-d35f-4333-b466-d8c2ece570e0") : secret "agc-cred-16d391637b1c" not found
Oct 30 04:51:58.874605 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:51:58.874561    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2907e444_12e2_47cd_b6d8_b5b4183ec275.slice/crio-fb1daf88ca02ca6873aa73cee3a50cdaf40a9984f41c5ba69a56c0f4804fcb29 WatchSource:0}: Error finding container fb1daf88ca02ca6873aa73cee3a50cdaf40a9984f41c5ba69a56c0f4804fcb29: Status 404 returned error can't find the container with id fb1daf88ca02ca6873aa73cee3a50cdaf40a9984f41c5ba69a56c0f4804fcb29
Oct 30 04:51:58.875287 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:58.875256    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"]
Oct 30 04:51:59.189657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:51:59.189622    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8" event={"ID":"2907e444-12e2-47cd-b6d8-b5b4183ec275","Type":"ContainerStarted","Data":"fb1daf88ca02ca6873aa73cee3a50cdaf40a9984f41c5ba69a56c0f4804fcb29"}
Oct 30 04:52:00.199603 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:00.199568    5072 generic.go:356] "Generic (PLEG): container finished" podID="2907e444-12e2-47cd-b6d8-b5b4183ec275" containerID="2970de22336dd61257653a01a55994d499a4ffa022e915d96ca536507ca58069" exitCode=0
Oct 30 04:52:00.200030 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:00.199627    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8" event={"ID":"2907e444-12e2-47cd-b6d8-b5b4183ec275","Type":"ContainerDied","Data":"2970de22336dd61257653a01a55994d499a4ffa022e915d96ca536507ca58069"}
Oct 30 04:52:00.789922 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:00.789891    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-ed8266/agc-cred-16d391637b1c: secret "agc-cred-16d391637b1c" not found
Oct 30 04:52:00.789922 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:00.789916    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87: secret "agc-cred-16d391637b1c" not found
Oct 30 04:52:00.790173 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:00.789985    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials podName:93374e47-d35f-4333-b466-d8c2ece570e0 nodeName:}" failed. No retries permitted until 2026-10-30 04:52:04.789968938 +0000 UTC m=+3304.531392019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87" (UID: "93374e47-d35f-4333-b466-d8c2ece570e0") : secret "agc-cred-16d391637b1c" not found
Oct 30 04:52:00.866741 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:00.866714    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:52:00.883019 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:00.882997    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:52:01.366711 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:01.366687    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:52:01.396951 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:01.396922    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-xks-ca\") pod \"2907e444-12e2-47cd-b6d8-b5b4183ec275\" (UID: \"2907e444-12e2-47cd-b6d8-b5b4183ec275\") "
Oct 30 04:52:01.397112 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:01.396988    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2907e444-12e2-47cd-b6d8-b5b4183ec275-kube-api-access-dwd7k\" (UniqueName: \"kubernetes.io/projected/2907e444-12e2-47cd-b6d8-b5b4183ec275-kube-api-access-dwd7k\") pod \"2907e444-12e2-47cd-b6d8-b5b4183ec275\" (UID: \"2907e444-12e2-47cd-b6d8-b5b4183ec275\") "
Oct 30 04:52:01.397112 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:01.397091    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-openshift-ca\") pod \"2907e444-12e2-47cd-b6d8-b5b4183ec275\" (UID: \"2907e444-12e2-47cd-b6d8-b5b4183ec275\") "
Oct 30 04:52:01.397490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:01.397458    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-openshift-ca" pod "2907e444-12e2-47cd-b6d8-b5b4183ec275" (UID: "2907e444-12e2-47cd-b6d8-b5b4183ec275"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:01.397783 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:01.397760    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:01.399381 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:01.399359    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-xks-ca" pod "2907e444-12e2-47cd-b6d8-b5b4183ec275" (UID: "2907e444-12e2-47cd-b6d8-b5b4183ec275"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:52:01.399454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:01.399406    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2907e444-12e2-47cd-b6d8-b5b4183ec275-kube-api-access-dwd7k" pod "2907e444-12e2-47cd-b6d8-b5b4183ec275" (UID: "2907e444-12e2-47cd-b6d8-b5b4183ec275"). InnerVolumeSpecName "kube-api-access-dwd7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:01.499106 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:01.498993    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2907e444-12e2-47cd-b6d8-b5b4183ec275-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:01.499106 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:01.499024    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwd7k\" (UniqueName: \"kubernetes.io/projected/2907e444-12e2-47cd-b6d8-b5b4183ec275-kube-api-access-dwd7k\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:01.678478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:01.678438    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6" podUID="4a5aa91f-09e1-45e5-abe7-f0c9575c2139" containerName="payload-processing" containerID="cri-o://3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d" gracePeriod=25
Oct 30 04:52:01.937867 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:01.937830    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:52:02.005559 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.005526    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-kube-api-access-gkgdp\" (UniqueName: \"kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-kube-api-access-gkgdp\") pod \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") "
Oct 30 04:52:02.005741 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.005724    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-provider-credentials\") pod \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") "
Oct 30 04:52:02.005796 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.005756    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-plugins-config-volume\") pod \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") "
Oct 30 04:52:02.005842 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.005825    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-routing-overlay-volume\") pod \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\" (UID: \"4a5aa91f-09e1-45e5-abe7-f0c9575c2139\") "
Oct 30 04:52:02.006349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.006316    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-plugins-config-volume" pod "4a5aa91f-09e1-45e5-abe7-f0c9575c2139" (UID: "4a5aa91f-09e1-45e5-abe7-f0c9575c2139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:02.006664 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.006406    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:02.006664 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.006406    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-routing-overlay-volume" pod "4a5aa91f-09e1-45e5-abe7-f0c9575c2139" (UID: "4a5aa91f-09e1-45e5-abe7-f0c9575c2139"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:02.008092 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.008052    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-provider-credentials" pod "4a5aa91f-09e1-45e5-abe7-f0c9575c2139" (UID: "4a5aa91f-09e1-45e5-abe7-f0c9575c2139"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:02.008425 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.008394    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-kube-api-access-gkgdp" pod "4a5aa91f-09e1-45e5-abe7-f0c9575c2139" (UID: "4a5aa91f-09e1-45e5-abe7-f0c9575c2139"). InnerVolumeSpecName "kube-api-access-gkgdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:02.108153 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.108027    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:02.108153 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.108059    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:02.108153 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.108105    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkgdp\" (UniqueName: \"kubernetes.io/projected/4a5aa91f-09e1-45e5-abe7-f0c9575c2139-kube-api-access-gkgdp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:02.215277 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.215247    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"
Oct 30 04:52:02.215436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.215269    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8" event={"ID":"2907e444-12e2-47cd-b6d8-b5b4183ec275","Type":"ContainerDied","Data":"fb1daf88ca02ca6873aa73cee3a50cdaf40a9984f41c5ba69a56c0f4804fcb29"}
Oct 30 04:52:02.215436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.215342    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb1daf88ca02ca6873aa73cee3a50cdaf40a9984f41c5ba69a56c0f4804fcb29"
Oct 30 04:52:02.216859 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.216830    5072 generic.go:356] "Generic (PLEG): container finished" podID="4a5aa91f-09e1-45e5-abe7-f0c9575c2139" containerID="3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d" exitCode=0
Oct 30 04:52:02.216987 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.216876    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"
Oct 30 04:52:02.216987 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.216913    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6" event={"ID":"4a5aa91f-09e1-45e5-abe7-f0c9575c2139","Type":"ContainerDied","Data":"3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d"}
Oct 30 04:52:02.216987 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.216967    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6" event={"ID":"4a5aa91f-09e1-45e5-abe7-f0c9575c2139","Type":"ContainerDied","Data":"33df459567f9e1ccb996ed8347a7968d96df7114ac68ca63d90de520b07ca63e"}
Oct 30 04:52:02.217165 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.216986    5072 scope.go:122] "RemoveContainer" containerID="3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d"
Oct 30 04:52:02.229138 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.229122    5072 scope.go:122] "RemoveContainer" containerID="3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d"
Oct 30 04:52:02.229407 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:02.229383    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d\": container with ID starting with 3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d not found: ID does not exist" containerID="3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d"
Oct 30 04:52:02.229524 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.229409    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d"} err="failed to get container status \"3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d\": rpc error: code = NotFound desc = could not find container \"3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d\": container with ID starting with 3641debdf4e9e88bd9426e95ad29027fddcb56e3447d2b38a32af76e7954a61d not found: ID does not exist"
Oct 30 04:52:02.274217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.274186    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"]
Oct 30 04:52:02.281044 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.281019    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-5f78pmrr6"]
Oct 30 04:52:02.410255 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:02.410176    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a5aa91f-09e1-45e5-abe7-f0c9575c2139" path="/var/lib/kubelet/pods/4a5aa91f-09e1-45e5-abe7-f0c9575c2139/volumes"
Oct 30 04:52:03.353333 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.353288    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp" podUID="b6d9c083-9be6-42fd-b059-a4a5791bc93d" containerName="payload-processing" containerID="cri-o://41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d" gracePeriod=25
Oct 30 04:52:03.625730 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.625666    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:52:03.729514 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.729486    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-provider-credentials\") pod \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") "
Oct 30 04:52:03.729707 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.729532    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-kube-api-access-t2zs2\" (UniqueName: \"kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-kube-api-access-t2zs2\") pod \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") "
Oct 30 04:52:03.729707 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.729658    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-plugins-config-volume\") pod \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") "
Oct 30 04:52:03.729841 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.729731    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-routing-overlay-volume\") pod \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\" (UID: \"b6d9c083-9be6-42fd-b059-a4a5791bc93d\") "
Oct 30 04:52:03.730117 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.730070    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-plugins-config-volume" pod "b6d9c083-9be6-42fd-b059-a4a5791bc93d" (UID: "b6d9c083-9be6-42fd-b059-a4a5791bc93d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:03.730240 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.730152    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-routing-overlay-volume" pod "b6d9c083-9be6-42fd-b059-a4a5791bc93d" (UID: "b6d9c083-9be6-42fd-b059-a4a5791bc93d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:03.731982 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.731959    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-kube-api-access-t2zs2" pod "b6d9c083-9be6-42fd-b059-a4a5791bc93d" (UID: "b6d9c083-9be6-42fd-b059-a4a5791bc93d"). InnerVolumeSpecName "kube-api-access-t2zs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:03.731982 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.731972    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-provider-credentials" pod "b6d9c083-9be6-42fd-b059-a4a5791bc93d" (UID: "b6d9c083-9be6-42fd-b059-a4a5791bc93d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:03.831882 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.831851    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:03.831882 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.831880    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:03.832043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.831894    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2zs2\" (UniqueName: \"kubernetes.io/projected/b6d9c083-9be6-42fd-b059-a4a5791bc93d-kube-api-access-t2zs2\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:03.832043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:03.831904    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9c083-9be6-42fd-b059-a4a5791bc93d-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:04.190565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.190524    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87" podUID="93374e47-d35f-4333-b466-d8c2ece570e0" containerName="payload-processing" containerID="cri-o://9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b" gracePeriod=25
Oct 30 04:52:04.235657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.235631    5072 generic.go:356] "Generic (PLEG): container finished" podID="b6d9c083-9be6-42fd-b059-a4a5791bc93d" containerID="41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d" exitCode=0
Oct 30 04:52:04.235753 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.235678    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"
Oct 30 04:52:04.235753 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.235712    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp" event={"ID":"b6d9c083-9be6-42fd-b059-a4a5791bc93d","Type":"ContainerDied","Data":"41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d"}
Oct 30 04:52:04.235910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.235763    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp" event={"ID":"b6d9c083-9be6-42fd-b059-a4a5791bc93d","Type":"ContainerDied","Data":"128724e3382430e95ca8656660257b759ea0d90181ca432437cc076a6e63a73f"}
Oct 30 04:52:04.235910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.235785    5072 scope.go:122] "RemoveContainer" containerID="41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d"
Oct 30 04:52:04.301493 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.301467    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"]
Oct 30 04:52:04.305681 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.305659    5072 scope.go:122] "RemoveContainer" containerID="41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d"
Oct 30 04:52:04.305968 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:04.305938    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d\": container with ID starting with 41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d not found: ID does not exist" containerID="41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d"
Oct 30 04:52:04.306104 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.305976    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d"} err="failed to get container status \"41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d\": rpc error: code = NotFound desc = could not find container \"41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d\": container with ID starting with 41816524227aafba352611b1bcd326f9e10aea810bddec108619aa6ed95d1d2d not found: ID does not exist"
Oct 30 04:52:04.312588 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.312566    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-5fbbgwhlp"]
Oct 30 04:52:04.412490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.412453    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d9c083-9be6-42fd-b059-a4a5791bc93d" path="/var/lib/kubelet/pods/b6d9c083-9be6-42fd-b059-a4a5791bc93d/volumes"
Oct 30 04:52:04.480761 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.480734    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:52:04.540278 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.540247    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials\") pod \"93374e47-d35f-4333-b466-d8c2ece570e0\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") "
Oct 30 04:52:04.540430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.540365    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-kube-api-access-ndlxx\" (UniqueName: \"kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-kube-api-access-ndlxx\") pod \"93374e47-d35f-4333-b466-d8c2ece570e0\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") "
Oct 30 04:52:04.540515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.540485    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-plugins-config-volume\") pod \"93374e47-d35f-4333-b466-d8c2ece570e0\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") "
Oct 30 04:52:04.540580 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.540550    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-routing-overlay-volume\") pod \"93374e47-d35f-4333-b466-d8c2ece570e0\" (UID: \"93374e47-d35f-4333-b466-d8c2ece570e0\") "
Oct 30 04:52:04.540967 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.540916    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-plugins-config-volume" pod "93374e47-d35f-4333-b466-d8c2ece570e0" (UID: "93374e47-d35f-4333-b466-d8c2ece570e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:04.541253 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.541070    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:04.541253 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.541048    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-routing-overlay-volume" pod "93374e47-d35f-4333-b466-d8c2ece570e0" (UID: "93374e47-d35f-4333-b466-d8c2ece570e0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:04.542774 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.542749    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials" pod "93374e47-d35f-4333-b466-d8c2ece570e0" (UID: "93374e47-d35f-4333-b466-d8c2ece570e0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:04.542970 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.542948    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-kube-api-access-ndlxx" pod "93374e47-d35f-4333-b466-d8c2ece570e0" (UID: "93374e47-d35f-4333-b466-d8c2ece570e0"). InnerVolumeSpecName "kube-api-access-ndlxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:04.642358 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.642326    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndlxx\" (UniqueName: \"kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-kube-api-access-ndlxx\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:04.642358 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.642356    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93374e47-d35f-4333-b466-d8c2ece570e0-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:04.642358 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:04.642366    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93374e47-d35f-4333-b466-d8c2ece570e0-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:05.245307 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:05.245269    5072 generic.go:356] "Generic (PLEG): container finished" podID="93374e47-d35f-4333-b466-d8c2ece570e0" containerID="9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b" exitCode=0
Oct 30 04:52:05.245478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:05.245321    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"
Oct 30 04:52:05.245478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:05.245356    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87" event={"ID":"93374e47-d35f-4333-b466-d8c2ece570e0","Type":"ContainerDied","Data":"9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b"}
Oct 30 04:52:05.245478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:05.245412    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87" event={"ID":"93374e47-d35f-4333-b466-d8c2ece570e0","Type":"ContainerDied","Data":"e2ac75650118aee39f7f1530d2b77b10e89f4b77e1f927c37fbd038f67f393a6"}
Oct 30 04:52:05.245478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:05.245445    5072 scope.go:122] "RemoveContainer" containerID="9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b"
Oct 30 04:52:05.260327 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:05.260303    5072 scope.go:122] "RemoveContainer" containerID="9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b"
Oct 30 04:52:05.260643 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:05.260623    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b\": container with ID starting with 9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b not found: ID does not exist" containerID="9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b"
Oct 30 04:52:05.260724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:05.260650    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b"} err="failed to get container status \"9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b\": rpc error: code = NotFound desc = could not find container \"9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b\": container with ID starting with 9017fb509547c13d1b9ddd4d6f6a8b51ee74c21495f998426795a0c550252d1b not found: ID does not exist"
Oct 30 04:52:05.305017 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:05.304992    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"]
Oct 30 04:52:05.317157 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:05.317134    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-6cdfzlt87"]
Oct 30 04:52:06.409695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:06.409659    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93374e47-d35f-4333-b466-d8c2ece570e0" path="/var/lib/kubelet/pods/93374e47-d35f-4333-b466-d8c2ece570e0/volumes"
Oct 30 04:52:11.757346 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:11.757284    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:52:13.499010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.498970    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"]
Oct 30 04:52:13.500534 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.500489    5072 cpu_manager.go:414] "containerMap: removing container" podUID="b6d9c083-9be6-42fd-b059-a4a5791bc93d" containerName="payload-processing"
Oct 30 04:52:13.500729 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.500711    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d9c083-9be6-42fd-b059-a4a5791bc93d" containerName="payload-processing"
Oct 30 04:52:13.500828 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.500817    5072 cpu_manager.go:414] "containerMap: removing container" podUID="93374e47-d35f-4333-b466-d8c2ece570e0" containerName="payload-processing"
Oct 30 04:52:13.500924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.500900    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93374e47-d35f-4333-b466-d8c2ece570e0" containerName="payload-processing"
Oct 30 04:52:13.501070 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.501057    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2907e444-12e2-47cd-b6d8-b5b4183ec275" containerName="revoke-keys"
Oct 30 04:52:13.501182 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.501168    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2907e444-12e2-47cd-b6d8-b5b4183ec275" containerName="revoke-keys"
Oct 30 04:52:13.501289 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.501279    5072 cpu_manager.go:414] "containerMap: removing container" podUID="4a5aa91f-09e1-45e5-abe7-f0c9575c2139" containerName="payload-processing"
Oct 30 04:52:13.501374 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.501363    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5aa91f-09e1-45e5-abe7-f0c9575c2139" containerName="payload-processing"
Oct 30 04:52:13.501665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.501643    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d9c083-9be6-42fd-b059-a4a5791bc93d" containerName="payload-processing"
Oct 30 04:52:13.501665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.501664    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="93374e47-d35f-4333-b466-d8c2ece570e0" containerName="payload-processing"
Oct 30 04:52:13.501843 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.501686    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2907e444-12e2-47cd-b6d8-b5b4183ec275" containerName="revoke-keys"
Oct 30 04:52:13.501843 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.501699    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5aa91f-09e1-45e5-abe7-f0c9575c2139" containerName="payload-processing"
Oct 30 04:52:13.508892 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.508867    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.534741 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.532284    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 04:52:13.534741 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.532372    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 04:52:13.534741 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.532443    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 04:52:13.538257 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.538225    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"]
Oct 30 04:52:13.542549 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.542481    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 04:52:13.552501 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.552459    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3befd60c1db0\""
Oct 30 04:52:13.553905 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.553884    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-ddlgx\""
Oct 30 04:52:13.658031 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.657987    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4vvt\" (UniqueName: \"kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-kube-api-access-z4vvt\") pod \"payload-processing-external-model-5f8949bcc5-q9m9b\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") " pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.658283 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.658161    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-routing-overlay-volume\") pod \"payload-processing-external-model-5f8949bcc5-q9m9b\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") " pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.658283 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.658222    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-provider-credentials\") pod \"payload-processing-external-model-5f8949bcc5-q9m9b\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") " pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.658413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.658341    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-plugins-config-volume\") pod \"payload-processing-external-model-5f8949bcc5-q9m9b\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") " pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.760612 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.760319    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-plugins-config-volume\") pod \"payload-processing-external-model-5f8949bcc5-q9m9b\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") " pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.760612 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.760478    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4vvt\" (UniqueName: \"kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-kube-api-access-z4vvt\") pod \"payload-processing-external-model-5f8949bcc5-q9m9b\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") " pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.760865 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.760633    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-routing-overlay-volume\") pod \"payload-processing-external-model-5f8949bcc5-q9m9b\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") " pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.760865 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.760695    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-provider-credentials\") pod \"payload-processing-external-model-5f8949bcc5-q9m9b\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") " pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.770109 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.762345    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-plugins-config-volume\") pod \"payload-processing-external-model-5f8949bcc5-q9m9b\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") " pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.770109 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.763471    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-routing-overlay-volume\") pod \"payload-processing-external-model-5f8949bcc5-q9m9b\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") " pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.794141 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.790911    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-provider-credentials\") pod \"payload-processing-external-model-5f8949bcc5-q9m9b\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") " pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.826876 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.826840    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4vvt\" (UniqueName: \"kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-kube-api-access-z4vvt\") pod \"payload-processing-external-model-5f8949bcc5-q9m9b\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") " pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:13.844005 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:13.838272    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:14.084353 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.081524    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"]
Oct 30 04:52:14.088292 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.088272    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.105271 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.105229    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"]
Oct 30 04:52:14.132892 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.121540    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3ad88dd1d8f2\""
Oct 30 04:52:14.172862 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.172744    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-routing-overlay-volume\") pod \"payload-processing-external-model-669d69b768-tlqsf\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") " pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.173277 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.173250    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ds74\" (UniqueName: \"kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-kube-api-access-5ds74\") pod \"payload-processing-external-model-669d69b768-tlqsf\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") " pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.173412 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.173358    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-plugins-config-volume\") pod \"payload-processing-external-model-669d69b768-tlqsf\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") " pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.173508 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.173491    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials\") pod \"payload-processing-external-model-669d69b768-tlqsf\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") " pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.233156 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:52:14.233122    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2208856_d6aa_4b7e_83df_0b659393770c.slice/crio-057110b30fe9dec06d8072060cccf847116f66ab1889f9f22bb578032ae766ae WatchSource:0}: Error finding container 057110b30fe9dec06d8072060cccf847116f66ab1889f9f22bb578032ae766ae: Status 404 returned error can't find the container with id 057110b30fe9dec06d8072060cccf847116f66ab1889f9f22bb578032ae766ae
Oct 30 04:52:14.233398 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.233372    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"]
Oct 30 04:52:14.274981 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.274946    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ds74\" (UniqueName: \"kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-kube-api-access-5ds74\") pod \"payload-processing-external-model-669d69b768-tlqsf\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") " pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.275227 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.275206    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-plugins-config-volume\") pod \"payload-processing-external-model-669d69b768-tlqsf\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") " pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.275396 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.275378    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials\") pod \"payload-processing-external-model-669d69b768-tlqsf\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") " pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.275644 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.275629    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-routing-overlay-volume\") pod \"payload-processing-external-model-669d69b768-tlqsf\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") " pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.278956 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.276331    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-plugins-config-volume\") pod \"payload-processing-external-model-669d69b768-tlqsf\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") " pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.279306 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.279283    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-routing-overlay-volume\") pod \"payload-processing-external-model-669d69b768-tlqsf\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") " pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.297395 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.296994    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials\") pod \"payload-processing-external-model-669d69b768-tlqsf\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") " pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.316663 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.315830    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ds74\" (UniqueName: \"kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-kube-api-access-5ds74\") pod \"payload-processing-external-model-669d69b768-tlqsf\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") " pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.366527 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.366487    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b" event={"ID":"e2208856-d6aa-4b7e-83df-0b659393770c","Type":"ContainerStarted","Data":"057110b30fe9dec06d8072060cccf847116f66ab1889f9f22bb578032ae766ae"}
Oct 30 04:52:14.411188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.411162    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:14.721372 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:52:14.721325    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d81d071_1a09_4344_b95a_d28e1eae7763.slice/crio-4ab68bda78f855b89d36e988b9e3cb8a6d8ee7808cea96b8861ca9fb98de053a WatchSource:0}: Error finding container 4ab68bda78f855b89d36e988b9e3cb8a6d8ee7808cea96b8861ca9fb98de053a: Status 404 returned error can't find the container with id 4ab68bda78f855b89d36e988b9e3cb8a6d8ee7808cea96b8861ca9fb98de053a
Oct 30 04:52:14.721810 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:14.721786    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"]
Oct 30 04:52:15.378025 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:15.377988    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf" event={"ID":"6d81d071-1a09-4344-b95a-d28e1eae7763","Type":"ContainerStarted","Data":"81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6"}
Oct 30 04:52:15.378339 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:15.378320    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:15.378441 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:15.378354    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf" event={"ID":"6d81d071-1a09-4344-b95a-d28e1eae7763","Type":"ContainerStarted","Data":"4ab68bda78f855b89d36e988b9e3cb8a6d8ee7808cea96b8861ca9fb98de053a"}
Oct 30 04:52:15.379959 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:15.379930    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b" event={"ID":"e2208856-d6aa-4b7e-83df-0b659393770c","Type":"ContainerStarted","Data":"8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650"}
Oct 30 04:52:15.380122 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:15.380060    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:15.422945 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:15.422886    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf" podStartSLOduration=1.42287326 podStartE2EDuration="1.42287326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:15.420316148 +0000 UTC m=+3315.161739246" watchObservedRunningTime="2026-10-30 04:52:15.42287326 +0000 UTC m=+3315.164296358"
Oct 30 04:52:15.517358 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:15.517303    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b" podStartSLOduration=2.5172893370000002 podStartE2EDuration="2.517289337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:15.515146319 +0000 UTC m=+3315.256569415" watchObservedRunningTime="2026-10-30 04:52:15.517289337 +0000 UTC m=+3315.258712455"
Oct 30 04:52:16.516447 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.516385    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"]
Oct 30 04:52:16.527396 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.527370    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.533831 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.533797    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"]
Oct 30 04:52:16.547006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.546981    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"agc-cred-74593f2e2965\""
Oct 30 04:52:16.547006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.547003    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"routing-overlay\""
Oct 30 04:52:16.557365 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.557316    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"kube-root-ca.crt\""
Oct 30 04:52:16.557365 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.557316    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ed8266\""
Oct 30 04:52:16.557365 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.557317    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"openshift-service-ca.crt\""
Oct 30 04:52:16.557565 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.557425    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"payload-processing-external-model-e2e--9d9c1ba6-dockercfg-rr45s\""
Oct 30 04:52:16.604293 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.604265    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpvkx\" (UniqueName: \"kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-kube-api-access-dpvkx\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.604469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.604377    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.604469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.604401    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.604588 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.604572    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.706349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.706311    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.706537 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.706397    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpvkx\" (UniqueName: \"kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-kube-api-access-dpvkx\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.706537 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.706482    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.706537 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.706510    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.707337 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.707304    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.707475 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.707378    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.709038 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.709018    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.732349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.732313    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpvkx\" (UniqueName: \"kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-kube-api-access-dpvkx\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:16.848798 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:16.848718    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:17.166016 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:52:17.165983    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f15185_8dba_42d5_b4fe_ba7a06f5b7cb.slice/crio-27657c5c8c91b8f40fb489121a73ca744888f52b50b2ef815b02b2d38985ac87 WatchSource:0}: Error finding container 27657c5c8c91b8f40fb489121a73ca744888f52b50b2ef815b02b2d38985ac87: Status 404 returned error can't find the container with id 27657c5c8c91b8f40fb489121a73ca744888f52b50b2ef815b02b2d38985ac87
Oct 30 04:52:17.166819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:17.166779    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"]
Oct 30 04:52:17.405017 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:17.404977    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq" event={"ID":"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb","Type":"ContainerStarted","Data":"d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52"}
Oct 30 04:52:17.405225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:17.405039    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq" event={"ID":"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb","Type":"ContainerStarted","Data":"27657c5c8c91b8f40fb489121a73ca744888f52b50b2ef815b02b2d38985ac87"}
Oct 30 04:52:17.405225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:17.405177    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:17.451238 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:17.451193    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq" podStartSLOduration=1.451178632 podStartE2EDuration="1.451178632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:17.447161324 +0000 UTC m=+3317.188584431" watchObservedRunningTime="2026-10-30 04:52:17.451178632 +0000 UTC m=+3317.192601730"
Oct 30 04:52:19.856384 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:19.856346    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"]
Oct 30 04:52:19.859949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:19.859923    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:24.363282 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.363249    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"]
Oct 30 04:52:24.368219 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.368199    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.387053 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.387026    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"]
Oct 30 04:52:24.398546 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.395793    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"agc-cred-0e012fe878e9\""
Oct 30 04:52:24.398546 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.396116    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"payload-processing-external-model-e2e--a5b23dec-dockercfg-cpjvv\""
Oct 30 04:52:24.398546 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.396207    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"routing-overlay\""
Oct 30 04:52:24.398546 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.396299    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"kube-root-ca.crt\""
Oct 30 04:52:24.398874 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.398587    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"openshift-service-ca.crt\""
Oct 30 04:52:24.486071 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.486042    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-91705f\""
Oct 30 04:52:24.511034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.511005    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.511237 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.511166    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.511325 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.511299    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.511563 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.511542    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv5f2\" (UniqueName: \"kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-kube-api-access-dv5f2\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.613169 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.613128    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.613393 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.613270    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.613452 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.613406    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv5f2\" (UniqueName: \"kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-kube-api-access-dv5f2\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.613452 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.613444    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.614055 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.614027    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.614225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.614059    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.616258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.616231    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.637790 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.637763    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv5f2\" (UniqueName: \"kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-kube-api-access-dv5f2\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.684604 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.684573    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:24.857500 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:24.857443    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b" podUID="e2208856-d6aa-4b7e-83df-0b659393770c" containerName="payload-processing" containerID="cri-o://8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650" gracePeriod=25
Oct 30 04:52:25.012758 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.012594    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"]
Oct 30 04:52:25.127770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.127714    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:25.222069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.222036    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-kube-api-access-z4vvt\" (UniqueName: \"kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-kube-api-access-z4vvt\") pod \"e2208856-d6aa-4b7e-83df-0b659393770c\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") "
Oct 30 04:52:25.222354 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.222326    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-routing-overlay-volume\") pod \"e2208856-d6aa-4b7e-83df-0b659393770c\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") "
Oct 30 04:52:25.222563 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.222487    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-provider-credentials\") pod \"e2208856-d6aa-4b7e-83df-0b659393770c\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") "
Oct 30 04:52:25.222699 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.222606    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-plugins-config-volume\") pod \"e2208856-d6aa-4b7e-83df-0b659393770c\" (UID: \"e2208856-d6aa-4b7e-83df-0b659393770c\") "
Oct 30 04:52:25.222876 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.222853    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-routing-overlay-volume" pod "e2208856-d6aa-4b7e-83df-0b659393770c" (UID: "e2208856-d6aa-4b7e-83df-0b659393770c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:25.223189 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.223165    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-plugins-config-volume" pod "e2208856-d6aa-4b7e-83df-0b659393770c" (UID: "e2208856-d6aa-4b7e-83df-0b659393770c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:25.223385 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.223369    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:25.223451 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.223391    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2208856-d6aa-4b7e-83df-0b659393770c-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:25.225115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.225057    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-kube-api-access-z4vvt" pod "e2208856-d6aa-4b7e-83df-0b659393770c" (UID: "e2208856-d6aa-4b7e-83df-0b659393770c"). InnerVolumeSpecName "kube-api-access-z4vvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:25.225517 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.225496    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-provider-credentials" pod "e2208856-d6aa-4b7e-83df-0b659393770c" (UID: "e2208856-d6aa-4b7e-83df-0b659393770c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:25.325553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.325513    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:25.325553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.325554    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4vvt\" (UniqueName: \"kubernetes.io/projected/e2208856-d6aa-4b7e-83df-0b659393770c-kube-api-access-z4vvt\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:25.470392 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.470360    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"]
Oct 30 04:52:25.471586 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.471546    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e2208856-d6aa-4b7e-83df-0b659393770c" containerName="payload-processing"
Oct 30 04:52:25.471586 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.471572    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2208856-d6aa-4b7e-83df-0b659393770c" containerName="payload-processing"
Oct 30 04:52:25.471796 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.471777    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2208856-d6aa-4b7e-83df-0b659393770c" containerName="payload-processing"
Oct 30 04:52:25.476471 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.476418    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6" event={"ID":"22a0ea5f-d2d0-480c-8061-8bd392203482","Type":"ContainerStarted","Data":"5e67224389c46a4b909a7d50db63aea00b7b0a3d1c164bac49abc26d440c4087"}
Oct 30 04:52:25.477595 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.476543    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.477595 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.476695    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:25.477595 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.476795    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6" event={"ID":"22a0ea5f-d2d0-480c-8061-8bd392203482","Type":"ContainerStarted","Data":"94f5905ddece3bdc62e5049f18e12458b6986e33b4a3e0684c6bb9352810de2d"}
Oct 30 04:52:25.477815 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.477699    5072 generic.go:356] "Generic (PLEG): container finished" podID="e2208856-d6aa-4b7e-83df-0b659393770c" containerID="8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650" exitCode=0
Oct 30 04:52:25.477815 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.477759    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b" event={"ID":"e2208856-d6aa-4b7e-83df-0b659393770c","Type":"ContainerDied","Data":"8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650"}
Oct 30 04:52:25.477815 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.477794    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b" event={"ID":"e2208856-d6aa-4b7e-83df-0b659393770c","Type":"ContainerDied","Data":"057110b30fe9dec06d8072060cccf847116f66ab1889f9f22bb578032ae766ae"}
Oct 30 04:52:25.479587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.478198    5072 scope.go:122] "RemoveContainer" containerID="8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650"
Oct 30 04:52:25.479587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.479555    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"
Oct 30 04:52:25.493353 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.492860    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"]
Oct 30 04:52:25.494399 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.494378    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"kube-root-ca.crt\""
Oct 30 04:52:25.494548 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.494389    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"routing-overlay\""
Oct 30 04:52:25.494662 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.494485    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"payload-processing-external-model-e2e--e5d834c2-dockercfg-bcxjw\""
Oct 30 04:52:25.494736 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.494518    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c0e964\""
Oct 30 04:52:25.494736 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.494548    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"agc-cred-d17fe97dadf9\""
Oct 30 04:52:25.494736 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.494584    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"openshift-service-ca.crt\""
Oct 30 04:52:25.499996 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.499977    5072 scope.go:122] "RemoveContainer" containerID="8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650"
Oct 30 04:52:25.500326 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:25.500300    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650\": container with ID starting with 8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650 not found: ID does not exist" containerID="8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650"
Oct 30 04:52:25.500448 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.500336    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650"} err="failed to get container status \"8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650\": rpc error: code = NotFound desc = could not find container \"8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650\": container with ID starting with 8d30261f8eb63720092c59f8073a5565ba5624f179da628ed02be58e638b5650 not found: ID does not exist"
Oct 30 04:52:25.528276 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.528242    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.528445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.528304    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.528445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.528349    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9j49\" (UniqueName: \"kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-kube-api-access-c9j49\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.528572 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.528552    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.557992 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.553396    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6" podStartSLOduration=1.5533739579999999 podStartE2EDuration="1.553373958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:25.530366854 +0000 UTC m=+3325.271789952" watchObservedRunningTime="2026-10-30 04:52:25.553373958 +0000 UTC m=+3325.294797056"
Oct 30 04:52:25.619127 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.618034    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"]
Oct 30 04:52:25.637107 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.626827    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f8949bcc5-q9m9b"]
Oct 30 04:52:25.637107 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.631105    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.637107 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.631157    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.637107 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.631201    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9j49\" (UniqueName: \"kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-kube-api-access-c9j49\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.637107 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.631326    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.637107 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.632482    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.637107 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.633028    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.637107 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.636539    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.664745 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.662508    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9j49\" (UniqueName: \"kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-kube-api-access-c9j49\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:25.809033 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:25.808937    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:26.156914 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:52:26.156883    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935c7cbe_183b_48aa_b6eb_2709cb8aabb5.slice/crio-d0512f4231932d4134f8caf6936d6a4c13821be6e5b0682bd27e266b1d4ee78d WatchSource:0}: Error finding container d0512f4231932d4134f8caf6936d6a4c13821be6e5b0682bd27e266b1d4ee78d: Status 404 returned error can't find the container with id d0512f4231932d4134f8caf6936d6a4c13821be6e5b0682bd27e266b1d4ee78d
Oct 30 04:52:26.160126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:26.158836    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"]
Oct 30 04:52:26.393695 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:26.393666    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:26.410928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:26.410847    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2208856-d6aa-4b7e-83df-0b659393770c" path="/var/lib/kubelet/pods/e2208856-d6aa-4b7e-83df-0b659393770c/volumes"
Oct 30 04:52:26.490603 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:26.490565    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv" event={"ID":"935c7cbe-183b-48aa-b6eb-2709cb8aabb5","Type":"ContainerStarted","Data":"e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9"}
Oct 30 04:52:26.491273 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:26.490628    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv" event={"ID":"935c7cbe-183b-48aa-b6eb-2709cb8aabb5","Type":"ContainerStarted","Data":"d0512f4231932d4134f8caf6936d6a4c13821be6e5b0682bd27e266b1d4ee78d"}
Oct 30 04:52:26.491273 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:26.490678    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:26.536229 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:26.536184    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv" podStartSLOduration=1.536169567 podStartE2EDuration="1.536169567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:26.533196105 +0000 UTC m=+3326.274619204" watchObservedRunningTime="2026-10-30 04:52:26.536169567 +0000 UTC m=+3326.277592702"
Oct 30 04:52:26.552559 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:26.552524    5072 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3ad88dd1d8f2: secret "agc-cred-3ad88dd1d8f2" not found
Oct 30 04:52:26.552559 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:26.552563    5072 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3befd60c1db0: secret "agc-cred-3befd60c1db0" not found
Oct 30 04:52:26.552804 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:26.552576    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-669d69b768-tlqsf: [secret "agc-cred-3ad88dd1d8f2" not found, secret "agc-cred-3befd60c1db0" not found]
Oct 30 04:52:26.552804 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:26.552636    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials podName:6d81d071-1a09-4344-b95a-d28e1eae7763 nodeName:}" failed. No retries permitted until 2026-10-30 04:52:27.052616037 +0000 UTC m=+3326.794039116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials") pod "payload-processing-external-model-669d69b768-tlqsf" (UID: "6d81d071-1a09-4344-b95a-d28e1eae7763") : [secret "agc-cred-3ad88dd1d8f2" not found, secret "agc-cred-3befd60c1db0" not found]
Oct 30 04:52:26.668317 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:26.668188    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"]
Oct 30 04:52:27.042714 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.042683    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"]
Oct 30 04:52:27.050006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.049702    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:27.055330 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.055019    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"]
Oct 30 04:52:27.067001 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.061624    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a747ef2-lqf5k"]
Oct 30 04:52:27.070228 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.070202    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"]
Oct 30 04:52:27.070588 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:27.070572    5072 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3ad88dd1d8f2: secret "agc-cred-3ad88dd1d8f2" not found
Oct 30 04:52:27.070668 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:27.070592    5072 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3befd60c1db0: secret "agc-cred-3befd60c1db0" not found
Oct 30 04:52:27.070668 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:27.070604    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-669d69b768-tlqsf: [secret "agc-cred-3ad88dd1d8f2" not found, secret "agc-cred-3befd60c1db0" not found]
Oct 30 04:52:27.070790 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:27.070679    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials podName:6d81d071-1a09-4344-b95a-d28e1eae7763 nodeName:}" failed. No retries permitted until 2026-10-30 04:52:28.070660009 +0000 UTC m=+3327.812083099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials") pod "payload-processing-external-model-669d69b768-tlqsf" (UID: "6d81d071-1a09-4344-b95a-d28e1eae7763") : [secret "agc-cred-3ad88dd1d8f2" not found, secret "agc-cred-3befd60c1db0" not found]
Oct 30 04:52:27.073768 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.073743    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:52:27.073876 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.073774    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:52:27.171578 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.171541    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d08d2931-s8kmt\" (UID: \"a330b420-dede-4577-865f-8d8a7fa944ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:27.171722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.171669    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq2zh\" (UniqueName: \"kubernetes.io/projected/a330b420-dede-4577-865f-8d8a7fa944ad-kube-api-access-lq2zh\") pod \"maas-api-revoke-sub-d08d2931-s8kmt\" (UID: \"a330b420-dede-4577-865f-8d8a7fa944ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:27.171777 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.171730    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d08d2931-s8kmt\" (UID: \"a330b420-dede-4577-865f-8d8a7fa944ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:27.273808 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.273766    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d08d2931-s8kmt\" (UID: \"a330b420-dede-4577-865f-8d8a7fa944ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:27.274022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.273851    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq2zh\" (UniqueName: \"kubernetes.io/projected/a330b420-dede-4577-865f-8d8a7fa944ad-kube-api-access-lq2zh\") pod \"maas-api-revoke-sub-d08d2931-s8kmt\" (UID: \"a330b420-dede-4577-865f-8d8a7fa944ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:27.274022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.273910    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d08d2931-s8kmt\" (UID: \"a330b420-dede-4577-865f-8d8a7fa944ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:27.275249 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.275221    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d08d2931-s8kmt\" (UID: \"a330b420-dede-4577-865f-8d8a7fa944ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:27.277666 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.277619    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d08d2931-s8kmt\" (UID: \"a330b420-dede-4577-865f-8d8a7fa944ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:27.312354 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.312278    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq2zh\" (UniqueName: \"kubernetes.io/projected/a330b420-dede-4577-865f-8d8a7fa944ad-kube-api-access-lq2zh\") pod \"maas-api-revoke-sub-d08d2931-s8kmt\" (UID: \"a330b420-dede-4577-865f-8d8a7fa944ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:27.371792 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.371757    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:27.685164 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:52:27.685130    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda330b420_dede_4577_865f_8d8a7fa944ad.slice/crio-dfdd117032dc752ddb5c9a4b1c77527b69564e388164a8817cb64145d6a789aa WatchSource:0}: Error finding container dfdd117032dc752ddb5c9a4b1c77527b69564e388164a8817cb64145d6a789aa: Status 404 returned error can't find the container with id dfdd117032dc752ddb5c9a4b1c77527b69564e388164a8817cb64145d6a789aa
Oct 30 04:52:27.685754 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:27.685703    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"]
Oct 30 04:52:28.085689 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:28.085654    5072 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3ad88dd1d8f2: secret "agc-cred-3ad88dd1d8f2" not found
Oct 30 04:52:28.085689 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:28.085681    5072 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3befd60c1db0: secret "agc-cred-3befd60c1db0" not found
Oct 30 04:52:28.085689 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:28.085691    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-669d69b768-tlqsf: [secret "agc-cred-3ad88dd1d8f2" not found, secret "agc-cred-3befd60c1db0" not found]
Oct 30 04:52:28.085989 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:28.085754    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials podName:6d81d071-1a09-4344-b95a-d28e1eae7763 nodeName:}" failed. No retries permitted until 2026-10-30 04:52:30.085738699 +0000 UTC m=+3329.827161789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials") pod "payload-processing-external-model-669d69b768-tlqsf" (UID: "6d81d071-1a09-4344-b95a-d28e1eae7763") : [secret "agc-cred-3ad88dd1d8f2" not found, secret "agc-cred-3befd60c1db0" not found]
Oct 30 04:52:28.415129 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:28.415030    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004ef81e-66c7-43df-8891-9d2b45e431e3" path="/var/lib/kubelet/pods/004ef81e-66c7-43df-8891-9d2b45e431e3/volumes"
Oct 30 04:52:28.417624 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:28.417600    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:52:28.516562 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:28.516525    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt" event={"ID":"a330b420-dede-4577-865f-8d8a7fa944ad","Type":"ContainerStarted","Data":"9b8efbbfd797559d0b1cda46c60e5124f4989b5782490666eeae08f2f69b1165"}
Oct 30 04:52:28.516785 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:28.516591    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt" event={"ID":"a330b420-dede-4577-865f-8d8a7fa944ad","Type":"ContainerStarted","Data":"dfdd117032dc752ddb5c9a4b1c77527b69564e388164a8817cb64145d6a789aa"}
Oct 30 04:52:28.562185 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:28.562124    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt" podStartSLOduration=1.5621044849999999 podStartE2EDuration="1.562104485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:28.557524117 +0000 UTC m=+3328.298947238" watchObservedRunningTime="2026-10-30 04:52:28.562104485 +0000 UTC m=+3328.303527582"
Oct 30 04:52:29.526721 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.526690    5072 generic.go:356] "Generic (PLEG): container finished" podID="a330b420-dede-4577-865f-8d8a7fa944ad" containerID="9b8efbbfd797559d0b1cda46c60e5124f4989b5782490666eeae08f2f69b1165" exitCode=0
Oct 30 04:52:29.527154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.526769    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt" event={"ID":"a330b420-dede-4577-865f-8d8a7fa944ad","Type":"ContainerDied","Data":"9b8efbbfd797559d0b1cda46c60e5124f4989b5782490666eeae08f2f69b1165"}
Oct 30 04:52:29.667351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.667309    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"]
Oct 30 04:52:29.672064 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.672033    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.689135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.688915    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"openshift-service-ca.crt\""
Oct 30 04:52:29.689135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.688988    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"routing-overlay\""
Oct 30 04:52:29.689135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.688986    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"payload-processing-external-model-plugins-e2e-extmod-a-05bee8\""
Oct 30 04:52:29.689135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.689032    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"kube-root-ca.crt\""
Oct 30 04:52:29.693067 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.693025    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"]
Oct 30 04:52:29.694760 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.694720    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"agc-cred-569a092ced16\""
Oct 30 04:52:29.701094 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.701056    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"payload-processing-external-model-e2e--545d7aee-dockercfg-cpcvf\""
Oct 30 04:52:29.813850 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.813751    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbl5l\" (UniqueName: \"kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-kube-api-access-pbl5l\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.813850 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.813813    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.814212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.813977    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.814212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.814045    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.915691 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.915653    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.915849 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.915829    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbl5l\" (UniqueName: \"kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-kube-api-access-pbl5l\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.915911 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.915856    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.915961 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.915912    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.916522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.916489    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.916766 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.916701    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.918663 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.918642    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.940322 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.940298    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbl5l\" (UniqueName: \"kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-kube-api-access-pbl5l\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:29.987450 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:29.987423    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:30.118837 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:30.118761    5072 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3ad88dd1d8f2: secret "agc-cred-3ad88dd1d8f2" not found
Oct 30 04:52:30.118837 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:30.118790    5072 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3befd60c1db0: secret "agc-cred-3befd60c1db0" not found
Oct 30 04:52:30.118837 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:30.118800    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-669d69b768-tlqsf: [secret "agc-cred-3ad88dd1d8f2" not found, secret "agc-cred-3befd60c1db0" not found]
Oct 30 04:52:30.119064 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:30.118858    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials podName:6d81d071-1a09-4344-b95a-d28e1eae7763 nodeName:}" failed. No retries permitted until 2026-10-30 04:52:34.118842851 +0000 UTC m=+3333.860265925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials") pod "payload-processing-external-model-669d69b768-tlqsf" (UID: "6d81d071-1a09-4344-b95a-d28e1eae7763") : [secret "agc-cred-3ad88dd1d8f2" not found, secret "agc-cred-3befd60c1db0" not found]
Oct 30 04:52:30.297503 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.297467    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"]
Oct 30 04:52:30.297931 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:52:30.297906    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76d2c03_205f_45d6_8355_c8f54e304664.slice/crio-1875dc30e6d21fff28c7b508787be838ad35417f3951eadc745d56349601edc4 WatchSource:0}: Error finding container 1875dc30e6d21fff28c7b508787be838ad35417f3951eadc745d56349601edc4: Status 404 returned error can't find the container with id 1875dc30e6d21fff28c7b508787be838ad35417f3951eadc745d56349601edc4
Oct 30 04:52:30.537285 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.537249    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw" event={"ID":"e76d2c03-205f-45d6-8355-c8f54e304664","Type":"ContainerStarted","Data":"39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff"}
Oct 30 04:52:30.537784 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.537304    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw" event={"ID":"e76d2c03-205f-45d6-8355-c8f54e304664","Type":"ContainerStarted","Data":"1875dc30e6d21fff28c7b508787be838ad35417f3951eadc745d56349601edc4"}
Oct 30 04:52:30.537784 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.537326    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:30.587358 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.581847    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw" podStartSLOduration=1.5818294800000001 podStartE2EDuration="1.58182948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:30.581538346 +0000 UTC m=+3330.322961443" watchObservedRunningTime="2026-10-30 04:52:30.58182948 +0000 UTC m=+3330.323252578"
Oct 30 04:52:30.740803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.740758    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"]
Oct 30 04:52:30.744254 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.744228    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:30.747953 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.747934    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:30.759100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.758800    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"]
Oct 30 04:52:30.771540 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.770997    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"routing-overlay\""
Oct 30 04:52:30.771540 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.771209    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"payload-processing-external-model-e2e--30ae2d69-dockercfg-vdh2k\""
Oct 30 04:52:30.771540 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.771292    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"kube-root-ca.crt\""
Oct 30 04:52:30.771540 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.771381    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"agc-cred-7b3035355963\""
Oct 30 04:52:30.771540 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.771445    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"openshift-service-ca.crt\""
Oct 30 04:52:30.771540 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.771499    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b195d0\""
Oct 30 04:52:30.938363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.931594    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-xks-ca\") pod \"a330b420-dede-4577-865f-8d8a7fa944ad\" (UID: \"a330b420-dede-4577-865f-8d8a7fa944ad\") "
Oct 30 04:52:30.938363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.931695    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-openshift-ca\") pod \"a330b420-dede-4577-865f-8d8a7fa944ad\" (UID: \"a330b420-dede-4577-865f-8d8a7fa944ad\") "
Oct 30 04:52:30.938363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.931760    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a330b420-dede-4577-865f-8d8a7fa944ad-kube-api-access-lq2zh\" (UniqueName: \"kubernetes.io/projected/a330b420-dede-4577-865f-8d8a7fa944ad-kube-api-access-lq2zh\") pod \"a330b420-dede-4577-865f-8d8a7fa944ad\" (UID: \"a330b420-dede-4577-865f-8d8a7fa944ad\") "
Oct 30 04:52:30.938363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.932065    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:30.938363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.932178    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt5tq\" (UniqueName: \"kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-kube-api-access-vt5tq\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:30.938363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.932269    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:30.938363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.932482    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:30.938363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.933312    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-openshift-ca" pod "a330b420-dede-4577-865f-8d8a7fa944ad" (UID: "a330b420-dede-4577-865f-8d8a7fa944ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:30.939003 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.938875    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-xks-ca" pod "a330b420-dede-4577-865f-8d8a7fa944ad" (UID: "a330b420-dede-4577-865f-8d8a7fa944ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:52:30.939892 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:30.939867    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a330b420-dede-4577-865f-8d8a7fa944ad-kube-api-access-lq2zh" pod "a330b420-dede-4577-865f-8d8a7fa944ad" (UID: "a330b420-dede-4577-865f-8d8a7fa944ad"). InnerVolumeSpecName "kube-api-access-lq2zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:31.036807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.034271    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:31.036807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.035245    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:31.036807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.036041    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:31.036807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.036107    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:31.036807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.036183    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt5tq\" (UniqueName: \"kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-kube-api-access-vt5tq\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:31.036807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.036300    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:31.037423 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.036918    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:31.037423 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.036939    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq2zh\" (UniqueName: \"kubernetes.io/projected/a330b420-dede-4577-865f-8d8a7fa944ad-kube-api-access-lq2zh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:31.037423 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.036955    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330b420-dede-4577-865f-8d8a7fa944ad-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:31.047165 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.044734    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:31.061403 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.061375    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt5tq\" (UniqueName: \"kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-kube-api-access-vt5tq\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:31.075050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.075011    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:31.428917 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:52:31.428883    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdeda8ee_5cd0_49ce_b344_f914565487df.slice/crio-57ad6c88348f36270550a2d7df26677973d7abaf7be8c126dc3ce0d4559b4535 WatchSource:0}: Error finding container 57ad6c88348f36270550a2d7df26677973d7abaf7be8c126dc3ce0d4559b4535: Status 404 returned error can't find the container with id 57ad6c88348f36270550a2d7df26677973d7abaf7be8c126dc3ce0d4559b4535
Oct 30 04:52:31.429634 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.429607    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"]
Oct 30 04:52:31.546635 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.546600    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"
Oct 30 04:52:31.547047 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.546663    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt" event={"ID":"a330b420-dede-4577-865f-8d8a7fa944ad","Type":"ContainerDied","Data":"dfdd117032dc752ddb5c9a4b1c77527b69564e388164a8817cb64145d6a789aa"}
Oct 30 04:52:31.547047 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.546722    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfdd117032dc752ddb5c9a4b1c77527b69564e388164a8817cb64145d6a789aa"
Oct 30 04:52:31.548606 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.548581    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr" event={"ID":"cdeda8ee-5cd0-49ce-b344-f914565487df","Type":"ContainerStarted","Data":"fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd"}
Oct 30 04:52:31.548717 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.548624    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr" event={"ID":"cdeda8ee-5cd0-49ce-b344-f914565487df","Type":"ContainerStarted","Data":"57ad6c88348f36270550a2d7df26677973d7abaf7be8c126dc3ce0d4559b4535"}
Oct 30 04:52:31.548717 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.548693    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:31.593433 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.593389    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr" podStartSLOduration=1.59337263 podStartE2EDuration="1.59337263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:31.589570156 +0000 UTC m=+3331.330993256" watchObservedRunningTime="2026-10-30 04:52:31.59337263 +0000 UTC m=+3331.334795729"
Oct 30 04:52:31.670539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.670505    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf" podUID="6d81d071-1a09-4344-b95a-d28e1eae7763" containerName="payload-processing" containerID="cri-o://81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6" gracePeriod=25
Oct 30 04:52:31.964174 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:31.964145    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:32.152743 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.152668    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-kube-api-access-5ds74\" (UniqueName: \"kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-kube-api-access-5ds74\") pod \"6d81d071-1a09-4344-b95a-d28e1eae7763\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") "
Oct 30 04:52:32.152743 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.152736    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-routing-overlay-volume\") pod \"6d81d071-1a09-4344-b95a-d28e1eae7763\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") "
Oct 30 04:52:32.152935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.152863    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-plugins-config-volume\") pod \"6d81d071-1a09-4344-b95a-d28e1eae7763\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") "
Oct 30 04:52:32.152935 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.152886    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials\") pod \"6d81d071-1a09-4344-b95a-d28e1eae7763\" (UID: \"6d81d071-1a09-4344-b95a-d28e1eae7763\") "
Oct 30 04:52:32.153204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.153177    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-routing-overlay-volume" pod "6d81d071-1a09-4344-b95a-d28e1eae7763" (UID: "6d81d071-1a09-4344-b95a-d28e1eae7763"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:32.153325 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.153301    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-plugins-config-volume" pod "6d81d071-1a09-4344-b95a-d28e1eae7763" (UID: "6d81d071-1a09-4344-b95a-d28e1eae7763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:32.153440 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.153425    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:32.153480 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.153443    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d81d071-1a09-4344-b95a-d28e1eae7763-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:32.155118 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.155093    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-kube-api-access-5ds74" pod "6d81d071-1a09-4344-b95a-d28e1eae7763" (UID: "6d81d071-1a09-4344-b95a-d28e1eae7763"). InnerVolumeSpecName "kube-api-access-5ds74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:32.155287 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.155269    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials" pod "6d81d071-1a09-4344-b95a-d28e1eae7763" (UID: "6d81d071-1a09-4344-b95a-d28e1eae7763"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:32.254901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.254863    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ds74\" (UniqueName: \"kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-kube-api-access-5ds74\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:32.254901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.254901    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d81d071-1a09-4344-b95a-d28e1eae7763-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:32.558100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.558029    5072 generic.go:356] "Generic (PLEG): container finished" podID="6d81d071-1a09-4344-b95a-d28e1eae7763" containerID="81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6" exitCode=0
Oct 30 04:52:32.558653 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.558107    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"
Oct 30 04:52:32.558653 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.558129    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf" event={"ID":"6d81d071-1a09-4344-b95a-d28e1eae7763","Type":"ContainerDied","Data":"81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6"}
Oct 30 04:52:32.558653 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.558195    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-669d69b768-tlqsf" event={"ID":"6d81d071-1a09-4344-b95a-d28e1eae7763","Type":"ContainerDied","Data":"4ab68bda78f855b89d36e988b9e3cb8a6d8ee7808cea96b8861ca9fb98de053a"}
Oct 30 04:52:32.558653 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.558225    5072 scope.go:122] "RemoveContainer" containerID="81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6"
Oct 30 04:52:32.570303 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.570286    5072 scope.go:122] "RemoveContainer" containerID="81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6"
Oct 30 04:52:32.570580 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:32.570561    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6\": container with ID starting with 81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6 not found: ID does not exist" containerID="81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6"
Oct 30 04:52:32.570634 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.570588    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6"} err="failed to get container status \"81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6\": rpc error: code = NotFound desc = could not find container \"81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6\": container with ID starting with 81f4d67e6109a16e279765125f7e73abefcb2fb262593c96eb9adb9a53d17dc6 not found: ID does not exist"
Oct 30 04:52:32.613687 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.613662    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"]
Oct 30 04:52:32.626242 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:32.626208    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-669d69b768-tlqsf"]
Oct 30 04:52:33.190596 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.190557    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"]
Oct 30 04:52:33.191413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.191389    5072 cpu_manager.go:414] "containerMap: removing container" podUID="6d81d071-1a09-4344-b95a-d28e1eae7763" containerName="payload-processing"
Oct 30 04:52:33.191413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.191412    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d81d071-1a09-4344-b95a-d28e1eae7763" containerName="payload-processing"
Oct 30 04:52:33.191601 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.191452    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a330b420-dede-4577-865f-8d8a7fa944ad" containerName="revoke-keys"
Oct 30 04:52:33.191601 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.191461    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a330b420-dede-4577-865f-8d8a7fa944ad" containerName="revoke-keys"
Oct 30 04:52:33.191710 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.191652    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d81d071-1a09-4344-b95a-d28e1eae7763" containerName="payload-processing"
Oct 30 04:52:33.191710 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.191674    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a330b420-dede-4577-865f-8d8a7fa944ad" containerName="revoke-keys"
Oct 30 04:52:33.196862 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.196839    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.215553 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.212814    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c0e964\"/\"agc-cred-fefd25c3961a\""
Oct 30 04:52:33.218614 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.218588    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"]
Oct 30 04:52:33.368612 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.368576    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbtcv\" (UniqueName: \"kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-kube-api-access-xbtcv\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.368809 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.368742    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.368809 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.368800    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.368911 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.368841    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.470226 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.470138    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.470226 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.470195    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.470226 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.470211    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.470486 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.470310    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbtcv\" (UniqueName: \"kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-kube-api-access-xbtcv\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.470972 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.470946    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.471279 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.470948    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.473190 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.473164    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.494043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.494023    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbtcv\" (UniqueName: \"kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-kube-api-access-xbtcv\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.520271 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.520247    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:33.825932 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:52:33.825887    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8c0ab0_a99a_4116_b0fb_e381049a5cbf.slice/crio-1db2bbd0e6afe3245d1eb52ef15cf9d73e8d0e8ca46e3e51aad02437b69561b6 WatchSource:0}: Error finding container 1db2bbd0e6afe3245d1eb52ef15cf9d73e8d0e8ca46e3e51aad02437b69561b6: Status 404 returned error can't find the container with id 1db2bbd0e6afe3245d1eb52ef15cf9d73e8d0e8ca46e3e51aad02437b69561b6
Oct 30 04:52:33.826605 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:33.826579    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"]
Oct 30 04:52:34.408986 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:34.408948    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d81d071-1a09-4344-b95a-d28e1eae7763" path="/var/lib/kubelet/pods/6d81d071-1a09-4344-b95a-d28e1eae7763/volumes"
Oct 30 04:52:34.579034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:34.578994    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms" event={"ID":"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf","Type":"ContainerStarted","Data":"da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150"}
Oct 30 04:52:34.579223 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:34.579051    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms" event={"ID":"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf","Type":"ContainerStarted","Data":"1db2bbd0e6afe3245d1eb52ef15cf9d73e8d0e8ca46e3e51aad02437b69561b6"}
Oct 30 04:52:34.579223 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:34.579097    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:34.624506 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:34.624452    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms" podStartSLOduration=1.6244391519999999 podStartE2EDuration="1.624439152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:34.62134894 +0000 UTC m=+3334.362772050" watchObservedRunningTime="2026-10-30 04:52:34.624439152 +0000 UTC m=+3334.365862243"
Oct 30 04:52:37.504375 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:37.504348    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:37.504783 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:37.504437    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:40.008839 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.008803    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"]
Oct 30 04:52:40.015531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.015482    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.034995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.034962    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"]
Oct 30 04:52:40.046329 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.046291    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.046512 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.046376    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.046512 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.046414    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.046512 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.046468    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.046512 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.046503    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.046759 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.046554    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.046759 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.046597    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.046759 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.046667    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.046759 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.046691    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc5d7\" (UniqueName: \"kubernetes.io/projected/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-kube-api-access-bc5d7\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.118784 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.118750    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"]
Oct 30 04:52:40.124051 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.124021    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.140725 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.140689    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"agc-cred-b19d83c3194d\""
Oct 30 04:52:40.140885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.140762    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"payload-processing-external-model-e2e--18641857-dockercfg-h6j4z\""
Oct 30 04:52:40.140885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.140837    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"openshift-service-ca.crt\""
Oct 30 04:52:40.140885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.140838    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"routing-overlay\""
Oct 30 04:52:40.140885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.140873    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6378cc\""
Oct 30 04:52:40.141152 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.141006    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"kube-root-ca.crt\""
Oct 30 04:52:40.142170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.142149    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"]
Oct 30 04:52:40.147667 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.147642    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.147789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.147688    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.147789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.147760    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.147910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.147785    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc5d7\" (UniqueName: \"kubernetes.io/projected/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-kube-api-access-bc5d7\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.147910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.147859    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.147910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.147886    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.148061 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.147911    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9tg7\" (UniqueName: \"kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-kube-api-access-w9tg7\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.148061 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.148033    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.148211 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.148096    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.148272 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.148246    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.148334 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.148283    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.148392 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.148373    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.148462 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.148410    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.148517 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.148473    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.148780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.148753    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.148863 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.148762    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.148932 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.148879    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.149061 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.149036    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.171980 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.167653    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.177720 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.177599    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc5d7\" (UniqueName: \"kubernetes.io/projected/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-kube-api-access-bc5d7\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.179260 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.178229    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.180438 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.180408    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gl57g\" (UID: \"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.250436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.250398    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.250622 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.250524    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.250622 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.250552    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.250622 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:40.250560    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6378cc/agc-cred-b19d83c3194d: secret "agc-cred-b19d83c3194d" not found
Oct 30 04:52:40.250622 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.250577    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9tg7\" (UniqueName: \"kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-kube-api-access-w9tg7\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.250622 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:40.250583    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b: secret "agc-cred-b19d83c3194d" not found
Oct 30 04:52:40.250901 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:40.250646    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-provider-credentials podName:1aea4877-76c2-4b13-b086-d02b8f0fc8d3 nodeName:}" failed. No retries permitted until 2026-10-30 04:52:40.750625265 +0000 UTC m=+3340.492048359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b" (UID: "1aea4877-76c2-4b13-b086-d02b8f0fc8d3") : secret "agc-cred-b19d83c3194d" not found
Oct 30 04:52:40.251331 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.251312    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.251414 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.251312    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.277134 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.277047    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9tg7\" (UniqueName: \"kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-kube-api-access-w9tg7\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.291034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.291003    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"]
Oct 30 04:52:40.291399 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:40.291376    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b" podUID="1aea4877-76c2-4b13-b086-d02b8f0fc8d3"
Oct 30 04:52:40.332442 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.332407    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:40.640040 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.639963    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.662158 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.660232    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.663322 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.663295    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"]
Oct 30 04:52:40.665946 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.665895    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:52:40.666124 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.666010    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:52:40.666124 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.666058    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:52:40.758199 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.758163    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-plugins-config-volume\") pod \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") "
Oct 30 04:52:40.758387 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.758355    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-kube-api-access-w9tg7\" (UniqueName: \"kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-kube-api-access-w9tg7\") pod \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") "
Oct 30 04:52:40.758496 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.758479    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-routing-overlay-volume\") pod \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") "
Oct 30 04:52:40.758575 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.758538    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-plugins-config-volume" pod "1aea4877-76c2-4b13-b086-d02b8f0fc8d3" (UID: "1aea4877-76c2-4b13-b086-d02b8f0fc8d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:40.759151 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.758780    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-routing-overlay-volume" pod "1aea4877-76c2-4b13-b086-d02b8f0fc8d3" (UID: "1aea4877-76c2-4b13-b086-d02b8f0fc8d3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:40.759151 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.758954    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:40.759151 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:40.759118    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6378cc/agc-cred-b19d83c3194d: secret "agc-cred-b19d83c3194d" not found
Oct 30 04:52:40.759151 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:40.759137    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b: secret "agc-cred-b19d83c3194d" not found
Oct 30 04:52:40.759453 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:40.759194    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-provider-credentials podName:1aea4877-76c2-4b13-b086-d02b8f0fc8d3 nodeName:}" failed. No retries permitted until 2026-10-30 04:52:41.759175628 +0000 UTC m=+3341.500598718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b" (UID: "1aea4877-76c2-4b13-b086-d02b8f0fc8d3") : secret "agc-cred-b19d83c3194d" not found
Oct 30 04:52:40.759531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.759121    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:40.759591 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.759548    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:40.761316 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.761292    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-kube-api-access-w9tg7" pod "1aea4877-76c2-4b13-b086-d02b8f0fc8d3" (UID: "1aea4877-76c2-4b13-b086-d02b8f0fc8d3"). InnerVolumeSpecName "kube-api-access-w9tg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:40.861041 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:40.860995    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9tg7\" (UniqueName: \"kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-kube-api-access-w9tg7\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:41.552248 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:41.552216    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:52:41.681111 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:41.677249    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:41.681111 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:41.678936    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" event={"ID":"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0","Type":"ContainerStarted","Data":"a6cb426bd5ae7cf361502839d618165e9337fa38cd46ed4707ad27e62d187aaa"}
Oct 30 04:52:41.681111 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:41.678989    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" event={"ID":"8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0","Type":"ContainerStarted","Data":"755987cb4d966cf90428ba3e513e3a5c2cc8f084be55bf43bd4824e98e5138cb"}
Oct 30 04:52:41.794135 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:41.788050    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" podStartSLOduration=2.788029332 podStartE2EDuration="2.788029332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:41.753587155 +0000 UTC m=+3341.495010252" watchObservedRunningTime="2026-10-30 04:52:41.788029332 +0000 UTC m=+3341.529452430"
Oct 30 04:52:41.805243 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:41.805152    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b\" (UID: \"1aea4877-76c2-4b13-b086-d02b8f0fc8d3\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"
Oct 30 04:52:41.806513 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:41.806321    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6378cc/agc-cred-b19d83c3194d: object "ai-tenant-e2e-extmod-a-6378cc"/"agc-cred-b19d83c3194d" not registered
Oct 30 04:52:41.806693 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:41.806679    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b: object "ai-tenant-e2e-extmod-a-6378cc"/"agc-cred-b19d83c3194d" not registered
Oct 30 04:52:41.806850 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:41.806838    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-provider-credentials podName:1aea4877-76c2-4b13-b086-d02b8f0fc8d3 nodeName:}" failed. No retries permitted until 2026-10-30 04:52:43.806815149 +0000 UTC m=+3343.548238230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b" (UID: "1aea4877-76c2-4b13-b086-d02b8f0fc8d3") : object "ai-tenant-e2e-extmod-a-6378cc"/"agc-cred-b19d83c3194d" not registered
Oct 30 04:52:41.831125 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:41.827876    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"]
Oct 30 04:52:41.831487 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:41.831460    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:52:41.914129 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:41.912977    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95z5f8b"]
Oct 30 04:52:42.016523 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.016478    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1aea4877-76c2-4b13-b086-d02b8f0fc8d3-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:42.333299 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.333260    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:42.335316 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.335266    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gl57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 04:52:42.335500 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.335364    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" podUID="8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 04:52:42.420999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.418254    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aea4877-76c2-4b13-b086-d02b8f0fc8d3" path="/var/lib/kubelet/pods/1aea4877-76c2-4b13-b086-d02b8f0fc8d3/volumes"
Oct 30 04:52:42.579098 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.577187    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:52:42.749900 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.749867    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"]
Oct 30 04:52:42.776152 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.774484    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:42.786235 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.786162    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"]
Oct 30 04:52:42.792636 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.792611    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"agc-cred-7852b7404ccd\""
Oct 30 04:52:42.830504 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.830476    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:42.830702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.830533    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:42.830702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.830685    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:42.830801 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.830761    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xwvh\" (UniqueName: \"kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-kube-api-access-2xwvh\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:42.932495 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.932465    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:42.932789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.932552    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:42.932789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.932656    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:42.932789 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.932751    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xwvh\" (UniqueName: \"kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-kube-api-access-2xwvh\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:42.933258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.933230    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:42.933386 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.933325    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:42.935309 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.935287    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:42.984101 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:42.983751    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xwvh\" (UniqueName: \"kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-kube-api-access-2xwvh\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:43.096622 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:43.096517    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:43.333616 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:43.333581    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gl57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 04:52:43.333819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:43.333642    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" podUID="8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 04:52:43.443778 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:43.443747    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"]
Oct 30 04:52:43.443899 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:52:43.443758    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c9a69b_9bcb_4b1c_9697_6ce978ea4779.slice/crio-340f3e2508bd4a57d526e089cefd23180d53e9aed59e081a95790486d338f0ef WatchSource:0}: Error finding container 340f3e2508bd4a57d526e089cefd23180d53e9aed59e081a95790486d338f0ef: Status 404 returned error can't find the container with id 340f3e2508bd4a57d526e089cefd23180d53e9aed59e081a95790486d338f0ef
Oct 30 04:52:43.707928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:43.707837    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9" event={"ID":"75c9a69b-9bcb-4b1c-9697-6ce978ea4779","Type":"ContainerStarted","Data":"2b98fd88bda9c63b658f0e041e2abedd485384f3aeaabc046195bff2054b8745"}
Oct 30 04:52:43.707928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:43.707904    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9" event={"ID":"75c9a69b-9bcb-4b1c-9697-6ce978ea4779","Type":"ContainerStarted","Data":"340f3e2508bd4a57d526e089cefd23180d53e9aed59e081a95790486d338f0ef"}
Oct 30 04:52:43.708629 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:43.707953    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:43.989204 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:43.989012    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9" podStartSLOduration=1.988997714 podStartE2EDuration="1.988997714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:43.779475077 +0000 UTC m=+3343.520898175" watchObservedRunningTime="2026-10-30 04:52:43.988997714 +0000 UTC m=+3343.730420855"
Oct 30 04:52:43.991377 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:43.991347    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"]
Oct 30 04:52:43.996771 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:43.996749    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.020136 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.016734    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b195d0\"/\"agc-cred-d7cbf4c25914\""
Oct 30 04:52:44.020281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.020184    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"]
Oct 30 04:52:44.051160 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.051126    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.051291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.051213    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.051455 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.051436    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.051511 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.051495    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8chk\" (UniqueName: \"kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-kube-api-access-z8chk\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.153168 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.153137    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.153360 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.153189    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.153360 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.153345    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.153458 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.153388    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8chk\" (UniqueName: \"kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-kube-api-access-z8chk\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.154044 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.153978    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.154206 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.154113    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.175103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.173911    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.183018 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.182993    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8chk\" (UniqueName: \"kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-kube-api-access-z8chk\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.313239 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.313158    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:44.333111 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.333054    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gl57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 04:52:44.333230 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.333137    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" podUID="8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 04:52:44.643669 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:52:44.643639    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4688132f_c53c_4441_83a7_7d0947d554bc.slice/crio-ab9822746b7df32788c7858f3b295b3759e46c37cb8e690943b8d31d8dde17f8 WatchSource:0}: Error finding container ab9822746b7df32788c7858f3b295b3759e46c37cb8e690943b8d31d8dde17f8: Status 404 returned error can't find the container with id ab9822746b7df32788c7858f3b295b3759e46c37cb8e690943b8d31d8dde17f8
Oct 30 04:52:44.645174 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.645146    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"]
Oct 30 04:52:44.716737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:44.716705    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw" event={"ID":"4688132f-c53c-4441-83a7-7d0947d554bc","Type":"ContainerStarted","Data":"ab9822746b7df32788c7858f3b295b3759e46c37cb8e690943b8d31d8dde17f8"}
Oct 30 04:52:45.333400 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:45.333362    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gl57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 04:52:45.333585 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:45.333417    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" podUID="8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 04:52:45.591138 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:45.591016    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:52:45.730918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:45.730873    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw" event={"ID":"4688132f-c53c-4441-83a7-7d0947d554bc","Type":"ContainerStarted","Data":"7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151"}
Oct 30 04:52:45.731414 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:45.731049    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:45.789065 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:45.789030    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"]
Oct 30 04:52:45.790785 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:45.790742    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw" podStartSLOduration=2.79072899 podStartE2EDuration="2.79072899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:45.786607071 +0000 UTC m=+3345.528030167" watchObservedRunningTime="2026-10-30 04:52:45.79072899 +0000 UTC m=+3345.532152088"
Oct 30 04:52:46.332822 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:46.332783    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gl57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 04:52:46.333011 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:46.332840    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" podUID="8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 04:52:47.333454 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:47.333418    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gl57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 04:52:47.333859 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:47.333474    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" podUID="8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 04:52:48.333426 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:48.333392    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gl57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 04:52:48.333612 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:48.333453    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" podUID="8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 04:52:49.340640 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:49.340608    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:49.341228 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:49.341091    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:49.342218 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:49.342190    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g"
Oct 30 04:52:49.687334 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:49.687291    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"]
Oct 30 04:52:50.790407 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:50.790367    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv" podUID="935c7cbe-183b-48aa-b6eb-2709cb8aabb5" containerName="payload-processing" containerID="cri-o://e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9" gracePeriod=25
Oct 30 04:52:51.070538 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.070501    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:51.185190 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.185153    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-routing-overlay-volume\") pod \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") "
Oct 30 04:52:51.185368 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.185209    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-plugins-config-volume\") pod \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") "
Oct 30 04:52:51.185368 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.185336    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-kube-api-access-c9j49\" (UniqueName: \"kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-kube-api-access-c9j49\") pod \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") "
Oct 30 04:52:51.185522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.185476    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-provider-credentials\") pod \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\" (UID: \"935c7cbe-183b-48aa-b6eb-2709cb8aabb5\") "
Oct 30 04:52:51.185613 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.185587    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-routing-overlay-volume" pod "935c7cbe-183b-48aa-b6eb-2709cb8aabb5" (UID: "935c7cbe-183b-48aa-b6eb-2709cb8aabb5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:51.185685 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.185647    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-plugins-config-volume" pod "935c7cbe-183b-48aa-b6eb-2709cb8aabb5" (UID: "935c7cbe-183b-48aa-b6eb-2709cb8aabb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:51.186022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.185998    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:51.186022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.186025    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:51.187910 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.187880    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-kube-api-access-c9j49" pod "935c7cbe-183b-48aa-b6eb-2709cb8aabb5" (UID: "935c7cbe-183b-48aa-b6eb-2709cb8aabb5"). InnerVolumeSpecName "kube-api-access-c9j49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:51.188018 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.187923    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-provider-credentials" pod "935c7cbe-183b-48aa-b6eb-2709cb8aabb5" (UID: "935c7cbe-183b-48aa-b6eb-2709cb8aabb5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:51.286802 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.286766    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9j49\" (UniqueName: \"kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-kube-api-access-c9j49\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:51.286802 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.286799    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/935c7cbe-183b-48aa-b6eb-2709cb8aabb5-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:51.784336 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.784294    5072 generic.go:356] "Generic (PLEG): container finished" podID="935c7cbe-183b-48aa-b6eb-2709cb8aabb5" containerID="e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9" exitCode=0
Oct 30 04:52:51.784529 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.784372    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv" event={"ID":"935c7cbe-183b-48aa-b6eb-2709cb8aabb5","Type":"ContainerDied","Data":"e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9"}
Oct 30 04:52:51.784529 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.784433    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv" event={"ID":"935c7cbe-183b-48aa-b6eb-2709cb8aabb5","Type":"ContainerDied","Data":"d0512f4231932d4134f8caf6936d6a4c13821be6e5b0682bd27e266b1d4ee78d"}
Oct 30 04:52:51.784529 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.784466    5072 scope.go:122] "RemoveContainer" containerID="e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9"
Oct 30 04:52:51.784723 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.784642    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"
Oct 30 04:52:51.798828 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.798638    5072 scope.go:122] "RemoveContainer" containerID="e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9"
Oct 30 04:52:51.799066 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:51.798915    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9\": container with ID starting with e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9 not found: ID does not exist" containerID="e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9"
Oct 30 04:52:51.799066 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.798943    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9"} err="failed to get container status \"e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9\": rpc error: code = NotFound desc = could not find container \"e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9\": container with ID starting with e975dfa11ca06917e2c0a5c0438cfb8dd1c629263221ce1377662bfc69820be9 not found: ID does not exist"
Oct 30 04:52:51.841621 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.841583    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"]
Oct 30 04:52:51.854547 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:51.854523    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff86n2hlv"]
Oct 30 04:52:52.410661 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:52.410620    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935c7cbe-183b-48aa-b6eb-2709cb8aabb5" path="/var/lib/kubelet/pods/935c7cbe-183b-48aa-b6eb-2709cb8aabb5/volumes"
Oct 30 04:52:54.688625 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:54.688587    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6" podUID="22a0ea5f-d2d0-480c-8061-8bd392203482" containerName="payload-processing" containerID="cri-o://5e67224389c46a4b909a7d50db63aea00b7b0a3d1c164bac49abc26d440c4087" gracePeriod=25
Oct 30 04:52:54.811712 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:54.811683    5072 generic.go:356] "Generic (PLEG): container finished" podID="22a0ea5f-d2d0-480c-8061-8bd392203482" containerID="5e67224389c46a4b909a7d50db63aea00b7b0a3d1c164bac49abc26d440c4087" exitCode=0
Oct 30 04:52:54.811808 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:54.811761    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6" event={"ID":"22a0ea5f-d2d0-480c-8061-8bd392203482","Type":"ContainerDied","Data":"5e67224389c46a4b909a7d50db63aea00b7b0a3d1c164bac49abc26d440c4087"}
Oct 30 04:52:54.976368 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:54.976343    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:55.044858 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.044821    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-plugins-config-volume\") pod \"22a0ea5f-d2d0-480c-8061-8bd392203482\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") "
Oct 30 04:52:55.045038 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.044923    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-provider-credentials\" (UniqueName: \"kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-provider-credentials\") pod \"22a0ea5f-d2d0-480c-8061-8bd392203482\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") "
Oct 30 04:52:55.045038 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.044985    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-kube-api-access-dv5f2\" (UniqueName: \"kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-kube-api-access-dv5f2\") pod \"22a0ea5f-d2d0-480c-8061-8bd392203482\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") "
Oct 30 04:52:55.045038 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.045030    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-routing-overlay-volume\") pod \"22a0ea5f-d2d0-480c-8061-8bd392203482\" (UID: \"22a0ea5f-d2d0-480c-8061-8bd392203482\") "
Oct 30 04:52:55.045380 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.045350    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-plugins-config-volume" pod "22a0ea5f-d2d0-480c-8061-8bd392203482" (UID: "22a0ea5f-d2d0-480c-8061-8bd392203482"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:55.045841 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.045813    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:55.046155 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.045812    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-routing-overlay-volume" pod "22a0ea5f-d2d0-480c-8061-8bd392203482" (UID: "22a0ea5f-d2d0-480c-8061-8bd392203482"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:55.047834 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.047804    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-provider-credentials" pod "22a0ea5f-d2d0-480c-8061-8bd392203482" (UID: "22a0ea5f-d2d0-480c-8061-8bd392203482"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:55.048003 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.047987    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-kube-api-access-dv5f2" pod "22a0ea5f-d2d0-480c-8061-8bd392203482" (UID: "22a0ea5f-d2d0-480c-8061-8bd392203482"). InnerVolumeSpecName "kube-api-access-dv5f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:55.147583 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.147541    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:55.147583 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.147576    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv5f2\" (UniqueName: \"kubernetes.io/projected/22a0ea5f-d2d0-480c-8061-8bd392203482-kube-api-access-dv5f2\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:55.147836 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.147593    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22a0ea5f-d2d0-480c-8061-8bd392203482-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:52:55.733892 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.733861    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:52:55.830907 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.830874    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"]
Oct 30 04:52:55.831501 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.831464    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6" event={"ID":"22a0ea5f-d2d0-480c-8061-8bd392203482","Type":"ContainerDied","Data":"94f5905ddece3bdc62e5049f18e12458b6986e33b4a3e0684c6bb9352810de2d"}
Oct 30 04:52:55.831657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.831535    5072 scope.go:122] "RemoveContainer" containerID="5e67224389c46a4b909a7d50db63aea00b7b0a3d1c164bac49abc26d440c4087"
Oct 30 04:52:55.831657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.831546    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"
Oct 30 04:52:55.920463 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:52:55.920420    5072 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a0ea5f_d2d0_480c_8061_8bd392203482.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a0ea5f_d2d0_480c_8061_8bd392203482.slice/crio-94f5905ddece3bdc62e5049f18e12458b6986e33b4a3e0684c6bb9352810de2d\": RecentStats: unable to find data in memory cache]"
Oct 30 04:52:55.935373 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.935346    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"]
Oct 30 04:52:55.942556 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:55.942525    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448dvdg6"]
Oct 30 04:52:56.415500 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:56.415340    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a0ea5f-d2d0-480c-8061-8bd392203482" path="/var/lib/kubelet/pods/22a0ea5f-d2d0-480c-8061-8bd392203482/volumes"
Oct 30 04:52:57.748206 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:57.748178    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:52:57.825130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:52:57.825097    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"]
Oct 30 04:53:00.831564 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:00.831507    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw" podUID="e76d2c03-205f-45d6-8355-c8f54e304664" containerName="payload-processing" containerID="cri-o://39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff" gracePeriod=25
Oct 30 04:53:01.111525 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.111498    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:53:01.236805 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.236772    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-provider-credentials\") pod \"e76d2c03-205f-45d6-8355-c8f54e304664\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") "
Oct 30 04:53:01.236984 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.236848    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-plugins-config-volume\") pod \"e76d2c03-205f-45d6-8355-c8f54e304664\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") "
Oct 30 04:53:01.237057 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.237045    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-routing-overlay-volume\") pod \"e76d2c03-205f-45d6-8355-c8f54e304664\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") "
Oct 30 04:53:01.237150 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.237109    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-kube-api-access-pbl5l\" (UniqueName: \"kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-kube-api-access-pbl5l\") pod \"e76d2c03-205f-45d6-8355-c8f54e304664\" (UID: \"e76d2c03-205f-45d6-8355-c8f54e304664\") "
Oct 30 04:53:01.237357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.237332    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-plugins-config-volume" pod "e76d2c03-205f-45d6-8355-c8f54e304664" (UID: "e76d2c03-205f-45d6-8355-c8f54e304664"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:53:01.237453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.237386    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-routing-overlay-volume" pod "e76d2c03-205f-45d6-8355-c8f54e304664" (UID: "e76d2c03-205f-45d6-8355-c8f54e304664"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:53:01.237665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.237647    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:53:01.237665 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.237664    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e76d2c03-205f-45d6-8355-c8f54e304664-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:53:01.239562 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.239536    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-provider-credentials" pod "e76d2c03-205f-45d6-8355-c8f54e304664" (UID: "e76d2c03-205f-45d6-8355-c8f54e304664"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:53:01.239649 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.239573    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-kube-api-access-pbl5l" pod "e76d2c03-205f-45d6-8355-c8f54e304664" (UID: "e76d2c03-205f-45d6-8355-c8f54e304664"). InnerVolumeSpecName "kube-api-access-pbl5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:53:01.339291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.339203    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbl5l\" (UniqueName: \"kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-kube-api-access-pbl5l\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:53:01.339291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.339235    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e76d2c03-205f-45d6-8355-c8f54e304664-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:53:01.752520 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.752486    5072 scope.go:122] "RemoveContainer" containerID="614da05a521d035c6a9adc734d70c533c1c1e54a34c713d56c1ef01de397b5e3"
Oct 30 04:53:01.763702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.763680    5072 scope.go:122] "RemoveContainer" containerID="501d235cf8ce753991c34db69bcd4c25cce90e358a56c88effe6021d7a004832"
Oct 30 04:53:01.893230 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.893198    5072 generic.go:356] "Generic (PLEG): container finished" podID="e76d2c03-205f-45d6-8355-c8f54e304664" containerID="39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff" exitCode=0
Oct 30 04:53:01.893681 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.893245    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"
Oct 30 04:53:01.893681 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.893283    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw" event={"ID":"e76d2c03-205f-45d6-8355-c8f54e304664","Type":"ContainerDied","Data":"39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff"}
Oct 30 04:53:01.893681 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.893346    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw" event={"ID":"e76d2c03-205f-45d6-8355-c8f54e304664","Type":"ContainerDied","Data":"1875dc30e6d21fff28c7b508787be838ad35417f3951eadc745d56349601edc4"}
Oct 30 04:53:01.893681 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.893391    5072 scope.go:122] "RemoveContainer" containerID="39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff"
Oct 30 04:53:01.904226 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.904207    5072 scope.go:122] "RemoveContainer" containerID="39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff"
Oct 30 04:53:01.904497 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:53:01.904479    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff\": container with ID starting with 39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff not found: ID does not exist" containerID="39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff"
Oct 30 04:53:01.904561 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.904505    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff"} err="failed to get container status \"39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff\": rpc error: code = NotFound desc = could not find container \"39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff\": container with ID starting with 39c79308fc58e47d7633c5484f223824e49743cad73d6fcf0fb09835a9aa02ff not found: ID does not exist"
Oct 30 04:53:01.952100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.951697    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"]
Oct 30 04:53:01.966056 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:01.966026    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d65q9vsw"]
Oct 30 04:53:02.409581 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.409553    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76d2c03-205f-45d6-8355-c8f54e304664" path="/var/lib/kubelet/pods/e76d2c03-205f-45d6-8355-c8f54e304664/volumes"
Oct 30 04:53:02.826141 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.826045    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr" podUID="cdeda8ee-5cd0-49ce-b344-f914565487df" containerName="payload-processing" containerID="cri-o://fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd" gracePeriod=25
Oct 30 04:53:02.940876 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.940847    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"]
Oct 30 04:53:02.941656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.941641    5072 cpu_manager.go:414] "containerMap: removing container" podUID="22a0ea5f-d2d0-480c-8061-8bd392203482" containerName="payload-processing"
Oct 30 04:53:02.941702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.941659    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a0ea5f-d2d0-480c-8061-8bd392203482" containerName="payload-processing"
Oct 30 04:53:02.941702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.941668    5072 cpu_manager.go:414] "containerMap: removing container" podUID="935c7cbe-183b-48aa-b6eb-2709cb8aabb5" containerName="payload-processing"
Oct 30 04:53:02.941702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.941674    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935c7cbe-183b-48aa-b6eb-2709cb8aabb5" containerName="payload-processing"
Oct 30 04:53:02.941702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.941681    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e76d2c03-205f-45d6-8355-c8f54e304664" containerName="payload-processing"
Oct 30 04:53:02.941702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.941687    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76d2c03-205f-45d6-8355-c8f54e304664" containerName="payload-processing"
Oct 30 04:53:02.941857 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.941792    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a0ea5f-d2d0-480c-8061-8bd392203482" containerName="payload-processing"
Oct 30 04:53:02.941857 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.941798    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="935c7cbe-183b-48aa-b6eb-2709cb8aabb5" containerName="payload-processing"
Oct 30 04:53:02.941857 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.941806    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76d2c03-205f-45d6-8355-c8f54e304664" containerName="payload-processing"
Oct 30 04:53:02.946284 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.946265    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"
Oct 30 04:53:02.965003 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.964975    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"]
Oct 30 04:53:02.969877 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.969853    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5f1508\""
Oct 30 04:53:02.970047 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.969948    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5f1508-dockercfg-prmvq\""
Oct 30 04:53:02.991399 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.991358    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"]
Oct 30 04:53:02.997472 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:02.997444    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"
Oct 30 04:53:03.020100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.020030    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"]
Oct 30 04:53:03.111152 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.091532    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e141e106-199b-4c45-aee5-2b7492ed4fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx\" (UID: \"e141e106-199b-4c45-aee5-2b7492ed4fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"
Oct 30 04:53:03.143535 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.143499    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbbkr\" (UniqueName: \"kubernetes.io/projected/737a04c9-e996-4325-b019-7fab7930e0ac-kube-api-access-qbbkr\") pod \"payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb\" (UID: \"737a04c9-e996-4325-b019-7fab7930e0ac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"
Oct 30 04:53:03.143799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.143773    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfqgz\" (UniqueName: \"kubernetes.io/projected/e141e106-199b-4c45-aee5-2b7492ed4fb4-kube-api-access-hfqgz\") pod \"payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx\" (UID: \"e141e106-199b-4c45-aee5-2b7492ed4fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"
Oct 30 04:53:03.143988 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.143919    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737a04c9-e996-4325-b019-7fab7930e0ac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb\" (UID: \"737a04c9-e996-4325-b019-7fab7930e0ac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"
Oct 30 04:53:03.225754 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.225726    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:53:03.245549 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.245517    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-plugins-config-volume\") pod \"cdeda8ee-5cd0-49ce-b344-f914565487df\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") "
Oct 30 04:53:03.245720 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.245642    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-routing-overlay-volume\") pod \"cdeda8ee-5cd0-49ce-b344-f914565487df\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") "
Oct 30 04:53:03.245720 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.245672    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-kube-api-access-vt5tq\" (UniqueName: \"kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-kube-api-access-vt5tq\") pod \"cdeda8ee-5cd0-49ce-b344-f914565487df\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") "
Oct 30 04:53:03.245859 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.245783    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-provider-credentials\") pod \"cdeda8ee-5cd0-49ce-b344-f914565487df\" (UID: \"cdeda8ee-5cd0-49ce-b344-f914565487df\") "
Oct 30 04:53:03.245985 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.245963    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-plugins-config-volume" pod "cdeda8ee-5cd0-49ce-b344-f914565487df" (UID: "cdeda8ee-5cd0-49ce-b344-f914565487df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:53:03.246066 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.246043    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-routing-overlay-volume" pod "cdeda8ee-5cd0-49ce-b344-f914565487df" (UID: "cdeda8ee-5cd0-49ce-b344-f914565487df"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:53:03.246388 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.246280    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbbkr\" (UniqueName: \"kubernetes.io/projected/737a04c9-e996-4325-b019-7fab7930e0ac-kube-api-access-qbbkr\") pod \"payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb\" (UID: \"737a04c9-e996-4325-b019-7fab7930e0ac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"
Oct 30 04:53:03.246581 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.246559    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfqgz\" (UniqueName: \"kubernetes.io/projected/e141e106-199b-4c45-aee5-2b7492ed4fb4-kube-api-access-hfqgz\") pod \"payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx\" (UID: \"e141e106-199b-4c45-aee5-2b7492ed4fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"
Oct 30 04:53:03.246688 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.246613    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737a04c9-e996-4325-b019-7fab7930e0ac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb\" (UID: \"737a04c9-e996-4325-b019-7fab7930e0ac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"
Oct 30 04:53:03.246753 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.246708    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e141e106-199b-4c45-aee5-2b7492ed4fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx\" (UID: \"e141e106-199b-4c45-aee5-2b7492ed4fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"
Oct 30 04:53:03.246817 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.246803    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:53:03.246877 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.246823    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdeda8ee-5cd0-49ce-b344-f914565487df-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:53:03.247482 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.247460    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e141e106-199b-4c45-aee5-2b7492ed4fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx\" (UID: \"e141e106-199b-4c45-aee5-2b7492ed4fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"
Oct 30 04:53:03.247594 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.247570    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737a04c9-e996-4325-b019-7fab7930e0ac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb\" (UID: \"737a04c9-e996-4325-b019-7fab7930e0ac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"
Oct 30 04:53:03.248299 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.248276    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-kube-api-access-vt5tq" pod "cdeda8ee-5cd0-49ce-b344-f914565487df" (UID: "cdeda8ee-5cd0-49ce-b344-f914565487df"). InnerVolumeSpecName "kube-api-access-vt5tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:53:03.248438 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.248408    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-provider-credentials" pod "cdeda8ee-5cd0-49ce-b344-f914565487df" (UID: "cdeda8ee-5cd0-49ce-b344-f914565487df"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:53:03.271677 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.271647    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfqgz\" (UniqueName: \"kubernetes.io/projected/e141e106-199b-4c45-aee5-2b7492ed4fb4-kube-api-access-hfqgz\") pod \"payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx\" (UID: \"e141e106-199b-4c45-aee5-2b7492ed4fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"
Oct 30 04:53:03.274436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.274389    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbbkr\" (UniqueName: \"kubernetes.io/projected/737a04c9-e996-4325-b019-7fab7930e0ac-kube-api-access-qbbkr\") pod \"payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb\" (UID: \"737a04c9-e996-4325-b019-7fab7930e0ac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"
Oct 30 04:53:03.321378 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.321340    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"
Oct 30 04:53:03.348394 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.348360    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt5tq\" (UniqueName: \"kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-kube-api-access-vt5tq\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:53:03.348394 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.348395    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdeda8ee-5cd0-49ce-b344-f914565487df-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:53:03.563347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.563313    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"
Oct 30 04:53:03.642443 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.642408    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"]
Oct 30 04:53:03.646588 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:53:03.646553    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode141e106_199b_4c45_aee5_2b7492ed4fb4.slice/crio-bd857dcfd05019d84d74fe98b92a95652670411ba8d500faefa4501ba3c7b193 WatchSource:0}: Error finding container bd857dcfd05019d84d74fe98b92a95652670411ba8d500faefa4501ba3c7b193: Status 404 returned error can't find the container with id bd857dcfd05019d84d74fe98b92a95652670411ba8d500faefa4501ba3c7b193
Oct 30 04:53:03.870228 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.870202    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"]
Oct 30 04:53:03.873256 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:53:03.873223    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737a04c9_e996_4325_b019_7fab7930e0ac.slice/crio-9d8434e68fd983fe7f0fc79b12983b8d0ef953aeca6ac054d075f3ed7cc4130e WatchSource:0}: Error finding container 9d8434e68fd983fe7f0fc79b12983b8d0ef953aeca6ac054d075f3ed7cc4130e: Status 404 returned error can't find the container with id 9d8434e68fd983fe7f0fc79b12983b8d0ef953aeca6ac054d075f3ed7cc4130e
Oct 30 04:53:03.915708 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.915684    5072 generic.go:356] "Generic (PLEG): container finished" podID="cdeda8ee-5cd0-49ce-b344-f914565487df" containerID="fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd" exitCode=0
Oct 30 04:53:03.915815 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.915729    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"
Oct 30 04:53:03.915815 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.915759    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr" event={"ID":"cdeda8ee-5cd0-49ce-b344-f914565487df","Type":"ContainerDied","Data":"fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd"}
Oct 30 04:53:03.915815 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.915814    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr" event={"ID":"cdeda8ee-5cd0-49ce-b344-f914565487df","Type":"ContainerDied","Data":"57ad6c88348f36270550a2d7df26677973d7abaf7be8c126dc3ce0d4559b4535"}
Oct 30 04:53:03.915994 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.915841    5072 scope.go:122] "RemoveContainer" containerID="fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd"
Oct 30 04:53:03.917522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.917499    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx" event={"ID":"e141e106-199b-4c45-aee5-2b7492ed4fb4","Type":"ContainerStarted","Data":"5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747"}
Oct 30 04:53:03.917632 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.917543    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx" event={"ID":"e141e106-199b-4c45-aee5-2b7492ed4fb4","Type":"ContainerStarted","Data":"bd857dcfd05019d84d74fe98b92a95652670411ba8d500faefa4501ba3c7b193"}
Oct 30 04:53:03.917632 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.917619    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"
Oct 30 04:53:03.918912 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.918892    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb" event={"ID":"737a04c9-e996-4325-b019-7fab7930e0ac","Type":"ContainerStarted","Data":"9d8434e68fd983fe7f0fc79b12983b8d0ef953aeca6ac054d075f3ed7cc4130e"}
Oct 30 04:53:03.927937 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.927919    5072 scope.go:122] "RemoveContainer" containerID="fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd"
Oct 30 04:53:03.928404 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:53:03.928380    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd\": container with ID starting with fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd not found: ID does not exist" containerID="fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd"
Oct 30 04:53:03.928501 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.928414    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd"} err="failed to get container status \"fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd\": rpc error: code = NotFound desc = could not find container \"fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd\": container with ID starting with fa894af16ca2b972e8ad482a52213e84677ff76b3315a40ddff09c4584a10cbd not found: ID does not exist"
Oct 30 04:53:03.976298 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.976269    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"]
Oct 30 04:53:03.980802 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:03.980777    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f44lfqr"]
Oct 30 04:53:04.024430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:04.024373    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx" podStartSLOduration=2.024360495 podStartE2EDuration="2.024360495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:53:04.020858213 +0000 UTC m=+3363.762281309" watchObservedRunningTime="2026-10-30 04:53:04.024360495 +0000 UTC m=+3363.765783592"
Oct 30 04:53:04.409445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:04.409413    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdeda8ee-5cd0-49ce-b344-f914565487df" path="/var/lib/kubelet/pods/cdeda8ee-5cd0-49ce-b344-f914565487df/volumes"
Oct 30 04:53:04.930956 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:04.930924    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb" event={"ID":"737a04c9-e996-4325-b019-7fab7930e0ac","Type":"ContainerStarted","Data":"0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7"}
Oct 30 04:53:04.931174 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:04.931022    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"
Oct 30 04:53:04.974474 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:04.974428    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb" podStartSLOduration=2.97441033 podStartE2EDuration="2.97441033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:53:04.972322592 +0000 UTC m=+3364.713745713" watchObservedRunningTime="2026-10-30 04:53:04.97441033 +0000 UTC m=+3364.715833428"
Oct 30 04:53:11.753517 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:53:11.753433    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:53:14.935044 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:14.935013    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"
Oct 30 04:53:15.943371 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:15.943345    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"
Oct 30 04:53:19.740954 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.740909    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"]
Oct 30 04:53:19.741931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.741907    5072 cpu_manager.go:414] "containerMap: removing container" podUID="cdeda8ee-5cd0-49ce-b344-f914565487df" containerName="payload-processing"
Oct 30 04:53:19.742056 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.741934    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdeda8ee-5cd0-49ce-b344-f914565487df" containerName="payload-processing"
Oct 30 04:53:19.742222 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.742206    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdeda8ee-5cd0-49ce-b344-f914565487df" containerName="payload-processing"
Oct 30 04:53:19.747617 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.747596    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"
Oct 30 04:53:19.763399 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.763375    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"]
Oct 30 04:53:19.770806 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.770781    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-134dad\""
Oct 30 04:53:19.770917 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.770866    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-134dad-dockercfg-nz9rd\""
Oct 30 04:53:19.837276 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.837235    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"]
Oct 30 04:53:19.844281 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.844249    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"
Oct 30 04:53:19.891148 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.880552    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8hjt\" (UniqueName: \"kubernetes.io/projected/0fcdf75f-0261-4af4-b574-5048789081f6-kube-api-access-f8hjt\") pod \"payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw\" (UID: \"0fcdf75f-0261-4af4-b574-5048789081f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"
Oct 30 04:53:19.891148 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.883304    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fcdf75f-0261-4af4-b574-5048789081f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw\" (UID: \"0fcdf75f-0261-4af4-b574-5048789081f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"
Oct 30 04:53:19.891148 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.882392    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"]
Oct 30 04:53:19.984114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.984040    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8hjt\" (UniqueName: \"kubernetes.io/projected/0fcdf75f-0261-4af4-b574-5048789081f6-kube-api-access-f8hjt\") pod \"payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw\" (UID: \"0fcdf75f-0261-4af4-b574-5048789081f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"
Oct 30 04:53:19.984114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.984092    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbx9b\" (UniqueName: \"kubernetes.io/projected/fc63106e-36a8-498e-9807-3676ef0dcb1c-kube-api-access-wbx9b\") pod \"payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5\" (UID: \"fc63106e-36a8-498e-9807-3676ef0dcb1c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"
Oct 30 04:53:19.984370 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.984125    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fcdf75f-0261-4af4-b574-5048789081f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw\" (UID: \"0fcdf75f-0261-4af4-b574-5048789081f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"
Oct 30 04:53:19.984370 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.984149    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc63106e-36a8-498e-9807-3676ef0dcb1c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5\" (UID: \"fc63106e-36a8-498e-9807-3676ef0dcb1c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"
Oct 30 04:53:19.984823 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:19.984803    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fcdf75f-0261-4af4-b574-5048789081f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw\" (UID: \"0fcdf75f-0261-4af4-b574-5048789081f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"
Oct 30 04:53:20.008807 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:20.008738    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8hjt\" (UniqueName: \"kubernetes.io/projected/0fcdf75f-0261-4af4-b574-5048789081f6-kube-api-access-f8hjt\") pod \"payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw\" (UID: \"0fcdf75f-0261-4af4-b574-5048789081f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"
Oct 30 04:53:20.064004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:20.063977    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"
Oct 30 04:53:20.085633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:20.085601    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbx9b\" (UniqueName: \"kubernetes.io/projected/fc63106e-36a8-498e-9807-3676ef0dcb1c-kube-api-access-wbx9b\") pod \"payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5\" (UID: \"fc63106e-36a8-498e-9807-3676ef0dcb1c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"
Oct 30 04:53:20.085766 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:20.085659    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc63106e-36a8-498e-9807-3676ef0dcb1c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5\" (UID: \"fc63106e-36a8-498e-9807-3676ef0dcb1c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"
Oct 30 04:53:20.086616 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:20.086590    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc63106e-36a8-498e-9807-3676ef0dcb1c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5\" (UID: \"fc63106e-36a8-498e-9807-3676ef0dcb1c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"
Oct 30 04:53:20.111873 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:20.111840    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbx9b\" (UniqueName: \"kubernetes.io/projected/fc63106e-36a8-498e-9807-3676ef0dcb1c-kube-api-access-wbx9b\") pod \"payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5\" (UID: \"fc63106e-36a8-498e-9807-3676ef0dcb1c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"
Oct 30 04:53:20.180537 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:20.180164    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"
Oct 30 04:53:20.374155 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:53:20.374124    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fcdf75f_0261_4af4_b574_5048789081f6.slice/crio-d05e2d80d892971e9730504f73846a806ccca5e1a59b802d448e13129ecf4434 WatchSource:0}: Error finding container d05e2d80d892971e9730504f73846a806ccca5e1a59b802d448e13129ecf4434: Status 404 returned error can't find the container with id d05e2d80d892971e9730504f73846a806ccca5e1a59b802d448e13129ecf4434
Oct 30 04:53:20.375197 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:20.375168    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"]
Oct 30 04:53:20.494894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:20.494861    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"]
Oct 30 04:53:20.495805 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:53:20.495739    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc63106e_36a8_498e_9807_3676ef0dcb1c.slice/crio-cce7e83a94602ed8f61e2e342b8ff7e22dd84829ab127ac27dcd7ef4ea91f2a0 WatchSource:0}: Error finding container cce7e83a94602ed8f61e2e342b8ff7e22dd84829ab127ac27dcd7ef4ea91f2a0: Status 404 returned error can't find the container with id cce7e83a94602ed8f61e2e342b8ff7e22dd84829ab127ac27dcd7ef4ea91f2a0
Oct 30 04:53:21.067113 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:21.067043    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5" event={"ID":"fc63106e-36a8-498e-9807-3676ef0dcb1c","Type":"ContainerStarted","Data":"263c6e7f2faac6a3be51eb45b78b7594c175175f220e647bf726e3761a951ec6"}
Oct 30 04:53:21.067655 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:21.067119    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5" event={"ID":"fc63106e-36a8-498e-9807-3676ef0dcb1c","Type":"ContainerStarted","Data":"cce7e83a94602ed8f61e2e342b8ff7e22dd84829ab127ac27dcd7ef4ea91f2a0"}
Oct 30 04:53:21.067655 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:21.067148    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"
Oct 30 04:53:21.068928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:21.068902    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw" event={"ID":"0fcdf75f-0261-4af4-b574-5048789081f6","Type":"ContainerStarted","Data":"51f2cb80618a41001899b8ce1be4ec7a69df60fbb85b81588a4f9395f2738dbd"}
Oct 30 04:53:21.069033 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:21.068954    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw" event={"ID":"0fcdf75f-0261-4af4-b574-5048789081f6","Type":"ContainerStarted","Data":"d05e2d80d892971e9730504f73846a806ccca5e1a59b802d448e13129ecf4434"}
Oct 30 04:53:21.069033 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:21.068985    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"
Oct 30 04:53:21.111992 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:21.111946    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5" podStartSLOduration=2.111931969 podStartE2EDuration="2.111931969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:53:21.109691615 +0000 UTC m=+3380.851114714" watchObservedRunningTime="2026-10-30 04:53:21.111931969 +0000 UTC m=+3380.853355113"
Oct 30 04:53:21.152713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:21.152657    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw" podStartSLOduration=2.152641401 podStartE2EDuration="2.152641401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:53:21.151194054 +0000 UTC m=+3380.892617147" watchObservedRunningTime="2026-10-30 04:53:21.152641401 +0000 UTC m=+3380.894064499"
Oct 30 04:53:32.079593 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:32.079564    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"
Oct 30 04:53:32.079999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:53:32.079646    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"
Oct 30 04:53:41.782712 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:53:41.782652    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:54:01.841601 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:01.841572    5072 scope.go:122] "RemoveContainer" containerID="8a90b469c09242d9650c9e9e12390541e7b7805109a89ad58f77b5d07f4234a2"
Oct 30 04:54:11.755164 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:54:11.755128    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:54:14.038499 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:14.038462    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"]
Oct 30 04:54:14.050942 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:14.050911    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53ecfec9-kqf5b"]
Oct 30 04:54:14.410347 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:14.410271    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="028a8ab3-ae5c-49b8-a011-4cabe867d158" path="/var/lib/kubelet/pods/028a8ab3-ae5c-49b8-a011-4cabe867d158/volumes"
Oct 30 04:54:17.040563 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:17.040533    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"]
Oct 30 04:54:17.053146 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:17.053110    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af8e4a9-lm88p"]
Oct 30 04:54:18.032284 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:18.032245    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"]
Oct 30 04:54:18.044877 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:18.044842    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307e3ddc-vwctb"]
Oct 30 04:54:18.410414 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:18.410333    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5e1716-636c-4077-9b7f-ae125f61ad14" path="/var/lib/kubelet/pods/ae5e1716-636c-4077-9b7f-ae125f61ad14/volumes"
Oct 30 04:54:18.410755 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:18.410741    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb2a3585-7fd4-4142-afcd-1c8976cff9b4" path="/var/lib/kubelet/pods/fb2a3585-7fd4-4142-afcd-1c8976cff9b4/volumes"
Oct 30 04:54:29.030633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:29.030596    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"]
Oct 30 04:54:29.042358 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:29.042327    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-882ee0b6-wmlt9"]
Oct 30 04:54:30.410149 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:30.410113    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db39036c-9f85-4b27-b099-abec63d0cd54" path="/var/lib/kubelet/pods/db39036c-9f85-4b27-b099-abec63d0cd54/volumes"
Oct 30 04:54:32.218184 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.218152    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"]
Oct 30 04:54:32.223035 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.223013    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.251812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.251783    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"]
Oct 30 04:54:32.320918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.320888    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f9cd17-openshift-default-dockercfg-62rdj\""
Oct 30 04:54:32.412803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.412766    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-envoy\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.413241 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.413212    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-certs\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.413469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.413451    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-socket\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.413617 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.413599    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-credential-socket\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.413728 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.413715    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-podinfo\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.413909 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.413886    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-token\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.414161 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.414143    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa3c63a-eaf0-4aec-b463-adc66812e632-istiod-ca-cert\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.414299 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.414283    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpnxj\" (UniqueName: \"kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-kube-api-access-hpnxj\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.414408 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.414396    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-data\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.516046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.515948    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-certs\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.516046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.515990    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-socket\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.516046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.516019    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-credential-socket\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.516390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.516142    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-podinfo\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.516390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.516185    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-token\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.516390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.516240    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa3c63a-eaf0-4aec-b463-adc66812e632-istiod-ca-cert\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.516390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.516275    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpnxj\" (UniqueName: \"kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-kube-api-access-hpnxj\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.516390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.516309    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-data\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.516674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.516506    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-credential-socket\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.516674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.516613    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-envoy\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.516674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.516644    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-certs\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.517258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.517233    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-data\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.517450 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.517261    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa3c63a-eaf0-4aec-b463-adc66812e632-istiod-ca-cert\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.518155 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.518124    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-socket\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.522094 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.520923    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-podinfo\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.523955 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.523929    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-envoy\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.541805 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.541779    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-token\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.542124 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.542053    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpnxj\" (UniqueName: \"kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-kube-api-access-hpnxj\") pod \"e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") " pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.549445 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.549418    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:32.880839 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.880808    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"]
Oct 30 04:54:32.881089 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:54:32.881056    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa3c63a_eaf0_4aec_b463_adc66812e632.slice/crio-c7a9036065d57da0de4a7dc75b853819789cacc2b707d23376273f49e4bb1436 WatchSource:0}: Error finding container c7a9036065d57da0de4a7dc75b853819789cacc2b707d23376273f49e4bb1436: Status 404 returned error can't find the container with id c7a9036065d57da0de4a7dc75b853819789cacc2b707d23376273f49e4bb1436
Oct 30 04:54:32.883859 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.883826    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:54:32.883964 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.883892    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:54:32.883964 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:32.883919    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:54:33.658150 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:33.658109    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" event={"ID":"bfa3c63a-eaf0-4aec-b463-adc66812e632","Type":"ContainerStarted","Data":"6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd"}
Oct 30 04:54:33.658594 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:33.658175    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" event={"ID":"bfa3c63a-eaf0-4aec-b463-adc66812e632","Type":"ContainerStarted","Data":"c7a9036065d57da0de4a7dc75b853819789cacc2b707d23376273f49e4bb1436"}
Oct 30 04:54:33.711957 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:33.711907    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" podStartSLOduration=1.711892159 podStartE2EDuration="1.711892159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:54:33.707799313 +0000 UTC m=+3453.449222435" watchObservedRunningTime="2026-10-30 04:54:33.711892159 +0000 UTC m=+3453.453315256"
Oct 30 04:54:34.550503 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:34.550470    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:34.557548 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:34.557521    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:34.665589 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:34.665550    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:34.666681 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:34.666660    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:54:37.944300 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:37.944268    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"]
Oct 30 04:54:37.949368 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:37.949346    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"
Oct 30 04:54:37.967906 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:37.967878    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"]
Oct 30 04:54:37.979509 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:37.979480    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f9cd17\""
Oct 30 04:54:37.998331 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:37.998302    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f9cd17-dockercfg-lj7qn\""
Oct 30 04:54:38.061150 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.053031    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt\" (UID: \"2bf7fdb4-1e6b-4575-a470-e659bd8306ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"
Oct 30 04:54:38.061150 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.053267    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnx74\" (UniqueName: \"kubernetes.io/projected/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-kube-api-access-fnx74\") pod \"payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt\" (UID: \"2bf7fdb4-1e6b-4575-a470-e659bd8306ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"
Oct 30 04:54:38.137791 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.132845    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"]
Oct 30 04:54:38.146039 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.139352    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"
Oct 30 04:54:38.146039 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.143427    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"]
Oct 30 04:54:38.154651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.154354    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt\" (UID: \"2bf7fdb4-1e6b-4575-a470-e659bd8306ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"
Oct 30 04:54:38.154651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.154485    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnx74\" (UniqueName: \"kubernetes.io/projected/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-kube-api-access-fnx74\") pod \"payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt\" (UID: \"2bf7fdb4-1e6b-4575-a470-e659bd8306ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"
Oct 30 04:54:38.156240 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.155933    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt\" (UID: \"2bf7fdb4-1e6b-4575-a470-e659bd8306ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"
Oct 30 04:54:38.179455 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.179414    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnx74\" (UniqueName: \"kubernetes.io/projected/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-kube-api-access-fnx74\") pod \"payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt\" (UID: \"2bf7fdb4-1e6b-4575-a470-e659bd8306ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"
Oct 30 04:54:38.255716 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.255634    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpwrd\" (UniqueName: \"kubernetes.io/projected/476eab0b-9b8a-449a-a65d-5dadf7f4483b-kube-api-access-wpwrd\") pod \"payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc\" (UID: \"476eab0b-9b8a-449a-a65d-5dadf7f4483b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"
Oct 30 04:54:38.255891 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.255858    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476eab0b-9b8a-449a-a65d-5dadf7f4483b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc\" (UID: \"476eab0b-9b8a-449a-a65d-5dadf7f4483b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"
Oct 30 04:54:38.267235 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.267210    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"
Oct 30 04:54:38.357191 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.357153    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476eab0b-9b8a-449a-a65d-5dadf7f4483b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc\" (UID: \"476eab0b-9b8a-449a-a65d-5dadf7f4483b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"
Oct 30 04:54:38.357348 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.357249    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpwrd\" (UniqueName: \"kubernetes.io/projected/476eab0b-9b8a-449a-a65d-5dadf7f4483b-kube-api-access-wpwrd\") pod \"payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc\" (UID: \"476eab0b-9b8a-449a-a65d-5dadf7f4483b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"
Oct 30 04:54:38.358335 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.358284    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476eab0b-9b8a-449a-a65d-5dadf7f4483b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc\" (UID: \"476eab0b-9b8a-449a-a65d-5dadf7f4483b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"
Oct 30 04:54:38.387896 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.387789    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpwrd\" (UniqueName: \"kubernetes.io/projected/476eab0b-9b8a-449a-a65d-5dadf7f4483b-kube-api-access-wpwrd\") pod \"payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc\" (UID: \"476eab0b-9b8a-449a-a65d-5dadf7f4483b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"
Oct 30 04:54:38.469486 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.469452    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"
Oct 30 04:54:38.580457 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:54:38.580418    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf7fdb4_1e6b_4575_a470_e659bd8306ed.slice/crio-6ed6d40cf1729e97b77ea3dce6dcd579bd85f6f20cb0d043722fd1f52d473081 WatchSource:0}: Error finding container 6ed6d40cf1729e97b77ea3dce6dcd579bd85f6f20cb0d043722fd1f52d473081: Status 404 returned error can't find the container with id 6ed6d40cf1729e97b77ea3dce6dcd579bd85f6f20cb0d043722fd1f52d473081
Oct 30 04:54:38.590305 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.589530    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"]
Oct 30 04:54:38.717298 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.717263    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt" event={"ID":"2bf7fdb4-1e6b-4575-a470-e659bd8306ed","Type":"ContainerStarted","Data":"c0990b8667af174f479e81ee134f54c93276fcaf1af96f7c70b3fb7609653210"}
Oct 30 04:54:38.717479 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.717317    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt" event={"ID":"2bf7fdb4-1e6b-4575-a470-e659bd8306ed","Type":"ContainerStarted","Data":"6ed6d40cf1729e97b77ea3dce6dcd579bd85f6f20cb0d043722fd1f52d473081"}
Oct 30 04:54:38.717479 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.717370    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"
Oct 30 04:54:38.761545 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.761479    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt" podStartSLOduration=1.7614613540000001 podStartE2EDuration="1.761461354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:54:38.758624455 +0000 UTC m=+3458.500047553" watchObservedRunningTime="2026-10-30 04:54:38.761461354 +0000 UTC m=+3458.502884452"
Oct 30 04:54:38.778491 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:54:38.778452    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476eab0b_9b8a_449a_a65d_5dadf7f4483b.slice/crio-2e7bc915c68c80101a7f1d5fe4a64da5ea83dd30d1fab04b7acbeb03aec9c230 WatchSource:0}: Error finding container 2e7bc915c68c80101a7f1d5fe4a64da5ea83dd30d1fab04b7acbeb03aec9c230: Status 404 returned error can't find the container with id 2e7bc915c68c80101a7f1d5fe4a64da5ea83dd30d1fab04b7acbeb03aec9c230
Oct 30 04:54:38.778898 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:38.778874    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"]
Oct 30 04:54:39.727351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:39.727246    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc" event={"ID":"476eab0b-9b8a-449a-a65d-5dadf7f4483b","Type":"ContainerStarted","Data":"79419a5814aa013c46bac61324f1fc71cb99c415f641ebc74a66cc1f8abd4ba5"}
Oct 30 04:54:39.727351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:39.727309    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc" event={"ID":"476eab0b-9b8a-449a-a65d-5dadf7f4483b","Type":"ContainerStarted","Data":"2e7bc915c68c80101a7f1d5fe4a64da5ea83dd30d1fab04b7acbeb03aec9c230"}
Oct 30 04:54:39.727850 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:39.727434    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"
Oct 30 04:54:39.772213 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:39.772154    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc" podStartSLOduration=2.772137831 podStartE2EDuration="2.772137831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:54:39.768772468 +0000 UTC m=+3459.510195565" watchObservedRunningTime="2026-10-30 04:54:39.772137831 +0000 UTC m=+3459.513560928"
Oct 30 04:54:41.754551 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:54:41.754513    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:54:49.730580 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:49.730551    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"
Oct 30 04:54:50.173211 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.173133    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"]
Oct 30 04:54:50.178061 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.178040    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:50.198031 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.197999    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"]
Oct 30 04:54:50.205704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.205677    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:54:50.205847 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.205731    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:54:50.211826 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.211801    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pcdr\" (UniqueName: \"kubernetes.io/projected/9100522a-a086-4a10-8ae8-962c23ab0cc4-kube-api-access-7pcdr\") pod \"maas-api-revoke-sub-a4403c82-wmlz7\" (UID: \"9100522a-a086-4a10-8ae8-962c23ab0cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:50.211951 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.211865    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4403c82-wmlz7\" (UID: \"9100522a-a086-4a10-8ae8-962c23ab0cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:50.211951 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.211941    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4403c82-wmlz7\" (UID: \"9100522a-a086-4a10-8ae8-962c23ab0cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:50.313684 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.313653    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcdr\" (UniqueName: \"kubernetes.io/projected/9100522a-a086-4a10-8ae8-962c23ab0cc4-kube-api-access-7pcdr\") pod \"maas-api-revoke-sub-a4403c82-wmlz7\" (UID: \"9100522a-a086-4a10-8ae8-962c23ab0cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:50.313879 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.313725    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4403c82-wmlz7\" (UID: \"9100522a-a086-4a10-8ae8-962c23ab0cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:50.313879 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.313755    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4403c82-wmlz7\" (UID: \"9100522a-a086-4a10-8ae8-962c23ab0cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:50.314519 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.314493    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4403c82-wmlz7\" (UID: \"9100522a-a086-4a10-8ae8-962c23ab0cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:50.316387 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.316368    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4403c82-wmlz7\" (UID: \"9100522a-a086-4a10-8ae8-962c23ab0cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:50.338235 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.338206    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pcdr\" (UniqueName: \"kubernetes.io/projected/9100522a-a086-4a10-8ae8-962c23ab0cc4-kube-api-access-7pcdr\") pod \"maas-api-revoke-sub-a4403c82-wmlz7\" (UID: \"9100522a-a086-4a10-8ae8-962c23ab0cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:50.494256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.494226    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:50.738302 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.738274    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"
Oct 30 04:54:50.817723 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:50.817546    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"]
Oct 30 04:54:51.856513 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:51.856478    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7" event={"ID":"9100522a-a086-4a10-8ae8-962c23ab0cc4","Type":"ContainerStarted","Data":"52833a15d3e59dbf6bd4f3fe6e47514925eb5bb57fee8853b7c0715f45d6c401"}
Oct 30 04:54:51.857002 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:51.856982    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7" event={"ID":"9100522a-a086-4a10-8ae8-962c23ab0cc4","Type":"ContainerStarted","Data":"9ef4634cdd361a99e9d374976338370300c3cbc641fe27ed1274458ca4d72fe5"}
Oct 30 04:54:51.904623 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:51.904565    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7" podStartSLOduration=1.904551638 podStartE2EDuration="1.904551638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:54:51.90170628 +0000 UTC m=+3471.643129381" watchObservedRunningTime="2026-10-30 04:54:51.904551638 +0000 UTC m=+3471.645974738"
Oct 30 04:54:52.866170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:52.866133    5072 generic.go:356] "Generic (PLEG): container finished" podID="9100522a-a086-4a10-8ae8-962c23ab0cc4" containerID="52833a15d3e59dbf6bd4f3fe6e47514925eb5bb57fee8853b7c0715f45d6c401" exitCode=0
Oct 30 04:54:52.866552 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:52.866218    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7" event={"ID":"9100522a-a086-4a10-8ae8-962c23ab0cc4","Type":"ContainerDied","Data":"52833a15d3e59dbf6bd4f3fe6e47514925eb5bb57fee8853b7c0715f45d6c401"}
Oct 30 04:54:53.044600 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.044570    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"]
Oct 30 04:54:53.049436 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.049414    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.065480 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.065453    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"agc-cred-3f798d461c2c\""
Oct 30 04:54:53.065618 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.065539    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"kube-root-ca.crt\""
Oct 30 04:54:53.065686 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.065650    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"payload-processing-external-model-plugins-e2e-extmod-b-483bea\""
Oct 30 04:54:53.065744 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.065730    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"agc-cred-349d99e30ae4\""
Oct 30 04:54:53.065806 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.065789    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"routing-overlay\""
Oct 30 04:54:53.065931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.065910    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"payload-processing-external-model-e2e--675c86f9-dockercfg-ftxjr\""
Oct 30 04:54:53.066154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.066138    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"openshift-service-ca.crt\""
Oct 30 04:54:53.072868 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.072830    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"]
Oct 30 04:54:53.177929 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.177855    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjlbt\" (UniqueName: \"kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-kube-api-access-pjlbt\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.177929 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.177907    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.178136 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.178016    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.178200 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.178147    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.279114 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.279049    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.279280 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.279182    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.279377 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.279362    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjlbt\" (UniqueName: \"kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-kube-api-access-pjlbt\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.279453 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.279422    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.279844 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.279819    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.280140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.280120    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.281729 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.281710    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.303250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.303227    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjlbt\" (UniqueName: \"kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-kube-api-access-pjlbt\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.371687 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.371663    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.684315 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:54:53.684274    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbe6653_c77e_4408_b6f0_bd16127b2d6b.slice/crio-e5cd3368fe0ac3bdb3ee4b8690bad32fa09883bf0ddfb653fa03ef9b3a5b5db8 WatchSource:0}: Error finding container e5cd3368fe0ac3bdb3ee4b8690bad32fa09883bf0ddfb653fa03ef9b3a5b5db8: Status 404 returned error can't find the container with id e5cd3368fe0ac3bdb3ee4b8690bad32fa09883bf0ddfb653fa03ef9b3a5b5db8
Oct 30 04:54:53.684662 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.684638    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"]
Oct 30 04:54:53.867577 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.867540    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"]
Oct 30 04:54:53.872609 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.872579    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:53.876012 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.875985    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf" event={"ID":"bbbe6653-c77e-4408-b6f0-bd16127b2d6b","Type":"ContainerStarted","Data":"21a4fbcc2cac20a05a7c1e529a956dcd7434dbcc8ba9205a23f750bdffc72cf2"}
Oct 30 04:54:53.876173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.876026    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf" event={"ID":"bbbe6653-c77e-4408-b6f0-bd16127b2d6b","Type":"ContainerStarted","Data":"e5cd3368fe0ac3bdb3ee4b8690bad32fa09883bf0ddfb653fa03ef9b3a5b5db8"}
Oct 30 04:54:53.876173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.876047    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:54:53.888837 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.888810    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ed8266\"/\"agc-cred-944fd0d2d6d5\""
Oct 30 04:54:53.894860 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.894474    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"]
Oct 30 04:54:53.962444 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.962384    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf" podStartSLOduration=0.962365281 podStartE2EDuration="962.365281ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:54:53.956531281 +0000 UTC m=+3473.697954380" watchObservedRunningTime="2026-10-30 04:54:53.962365281 +0000 UTC m=+3473.703788378"
Oct 30 04:54:53.989315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.989276    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:53.989487 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.989426    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:53.989671 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.989638    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4f6x\" (UniqueName: \"kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-kube-api-access-l4f6x\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:53.990056 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:53.989894    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:54.026398 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.026373    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:54.090889 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.090862    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9100522a-a086-4a10-8ae8-962c23ab0cc4-kube-api-access-7pcdr\" (UniqueName: \"kubernetes.io/projected/9100522a-a086-4a10-8ae8-962c23ab0cc4-kube-api-access-7pcdr\") pod \"9100522a-a086-4a10-8ae8-962c23ab0cc4\" (UID: \"9100522a-a086-4a10-8ae8-962c23ab0cc4\") "
Oct 30 04:54:54.091047 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.090932    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-openshift-ca\") pod \"9100522a-a086-4a10-8ae8-962c23ab0cc4\" (UID: \"9100522a-a086-4a10-8ae8-962c23ab0cc4\") "
Oct 30 04:54:54.091152 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.091112    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-xks-ca\") pod \"9100522a-a086-4a10-8ae8-962c23ab0cc4\" (UID: \"9100522a-a086-4a10-8ae8-962c23ab0cc4\") "
Oct 30 04:54:54.091258 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.091230    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-openshift-ca" pod "9100522a-a086-4a10-8ae8-962c23ab0cc4" (UID: "9100522a-a086-4a10-8ae8-962c23ab0cc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:54:54.091515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.091493    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4f6x\" (UniqueName: \"kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-kube-api-access-l4f6x\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:54.091691 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.091675    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:54.091768 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.091750    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:54.092019 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.091994    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:54.092275 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.092255    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:54:54.092909 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.092886    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:54.093012 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.092989    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:54.093573 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.093551    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9100522a-a086-4a10-8ae8-962c23ab0cc4-kube-api-access-7pcdr" pod "9100522a-a086-4a10-8ae8-962c23ab0cc4" (UID: "9100522a-a086-4a10-8ae8-962c23ab0cc4"). InnerVolumeSpecName "kube-api-access-7pcdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:54:54.093828 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.093813    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-xks-ca" pod "9100522a-a086-4a10-8ae8-962c23ab0cc4" (UID: "9100522a-a086-4a10-8ae8-962c23ab0cc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:54:54.094618 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.094602    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:54.115270 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.115249    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4f6x\" (UniqueName: \"kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-kube-api-access-l4f6x\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:54.192675 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.192650    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:54.193321 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.193302    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pcdr\" (UniqueName: \"kubernetes.io/projected/9100522a-a086-4a10-8ae8-962c23ab0cc4-kube-api-access-7pcdr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:54:54.193379 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.193325    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9100522a-a086-4a10-8ae8-962c23ab0cc4-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:54:54.501500 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:54:54.501468    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c75bb2_f1e9_4ef8_b536_73d8f299289a.slice/crio-0bc703caa31c74cee37876e5060cc41b0a298c84e076be35d8ff42fa928263db WatchSource:0}: Error finding container 0bc703caa31c74cee37876e5060cc41b0a298c84e076be35d8ff42fa928263db: Status 404 returned error can't find the container with id 0bc703caa31c74cee37876e5060cc41b0a298c84e076be35d8ff42fa928263db
Oct 30 04:54:54.502036 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.502010    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"]
Oct 30 04:54:54.897254 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.897168    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"
Oct 30 04:54:54.897841 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.897176    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7" event={"ID":"9100522a-a086-4a10-8ae8-962c23ab0cc4","Type":"ContainerDied","Data":"9ef4634cdd361a99e9d374976338370300c3cbc641fe27ed1274458ca4d72fe5"}
Oct 30 04:54:54.897971 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.897919    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ef4634cdd361a99e9d374976338370300c3cbc641fe27ed1274458ca4d72fe5"
Oct 30 04:54:54.899685 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.899651    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6" event={"ID":"67c75bb2-f1e9-4ef8-b536-73d8f299289a","Type":"ContainerStarted","Data":"b70b9aade521c4190cc337bac4824dea17d72eedb9f9e59ae66af7553a60b88f"}
Oct 30 04:54:54.899818 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.899711    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6" event={"ID":"67c75bb2-f1e9-4ef8-b536-73d8f299289a","Type":"ContainerStarted","Data":"0bc703caa31c74cee37876e5060cc41b0a298c84e076be35d8ff42fa928263db"}
Oct 30 04:54:54.899818 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.899754    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:54:54.944974 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:54.944921    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6" podStartSLOduration=1.944907742 podStartE2EDuration="1.944907742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:54:54.941582829 +0000 UTC m=+3474.683005927" watchObservedRunningTime="2026-10-30 04:54:54.944907742 +0000 UTC m=+3474.686330868"
Oct 30 04:54:57.487689 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.487655    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"]
Oct 30 04:54:57.488557 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.488541    5072 cpu_manager.go:414] "containerMap: removing container" podUID="9100522a-a086-4a10-8ae8-962c23ab0cc4" containerName="revoke-keys"
Oct 30 04:54:57.488557 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.488558    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9100522a-a086-4a10-8ae8-962c23ab0cc4" containerName="revoke-keys"
Oct 30 04:54:57.488680 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.488671    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9100522a-a086-4a10-8ae8-962c23ab0cc4" containerName="revoke-keys"
Oct 30 04:54:57.492837 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.492821    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"
Oct 30 04:54:57.515622 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.515600    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"]
Oct 30 04:54:57.531395 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.531368    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-82bec5\""
Oct 30 04:54:57.532780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.531742    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-82bec5-dockercfg-s2jnw\""
Oct 30 04:54:57.677842 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.677809    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66af17d7-50b8-4277-984a-a5fb20858c56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw\" (UID: \"66af17d7-50b8-4277-984a-a5fb20858c56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"
Oct 30 04:54:57.678050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.677849    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb95w\" (UniqueName: \"kubernetes.io/projected/66af17d7-50b8-4277-984a-a5fb20858c56-kube-api-access-qb95w\") pod \"payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw\" (UID: \"66af17d7-50b8-4277-984a-a5fb20858c56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"
Oct 30 04:54:57.779642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.779558    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66af17d7-50b8-4277-984a-a5fb20858c56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw\" (UID: \"66af17d7-50b8-4277-984a-a5fb20858c56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"
Oct 30 04:54:57.779642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.779596    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb95w\" (UniqueName: \"kubernetes.io/projected/66af17d7-50b8-4277-984a-a5fb20858c56-kube-api-access-qb95w\") pod \"payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw\" (UID: \"66af17d7-50b8-4277-984a-a5fb20858c56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"
Oct 30 04:54:57.780313 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.780289    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66af17d7-50b8-4277-984a-a5fb20858c56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw\" (UID: \"66af17d7-50b8-4277-984a-a5fb20858c56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"
Oct 30 04:54:57.804051 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.804019    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb95w\" (UniqueName: \"kubernetes.io/projected/66af17d7-50b8-4277-984a-a5fb20858c56-kube-api-access-qb95w\") pod \"payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw\" (UID: \"66af17d7-50b8-4277-984a-a5fb20858c56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"
Oct 30 04:54:57.807913 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:57.807891    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"
Oct 30 04:54:58.116765 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:54:58.116734    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66af17d7_50b8_4277_984a_a5fb20858c56.slice/crio-1815e0265f103ec4547f69afa259101a11567cf2bddd49014567f10397dca90b WatchSource:0}: Error finding container 1815e0265f103ec4547f69afa259101a11567cf2bddd49014567f10397dca90b: Status 404 returned error can't find the container with id 1815e0265f103ec4547f69afa259101a11567cf2bddd49014567f10397dca90b
Oct 30 04:54:58.117497 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:58.117474    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"]
Oct 30 04:54:58.944895 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:58.944863    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw" event={"ID":"66af17d7-50b8-4277-984a-a5fb20858c56","Type":"ContainerStarted","Data":"6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013"}
Oct 30 04:54:58.945297 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:58.944924    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw" event={"ID":"66af17d7-50b8-4277-984a-a5fb20858c56","Type":"ContainerStarted","Data":"1815e0265f103ec4547f69afa259101a11567cf2bddd49014567f10397dca90b"}
Oct 30 04:54:58.945297 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:58.944954    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"
Oct 30 04:54:58.991818 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:54:58.991766    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw" podStartSLOduration=1.991753477 podStartE2EDuration="1.991753477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:54:58.987810482 +0000 UTC m=+3478.729233578" watchObservedRunningTime="2026-10-30 04:54:58.991753477 +0000 UTC m=+3478.733176574"
Oct 30 04:55:01.052582 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.052543    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"]
Oct 30 04:55:01.058455 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.058434    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:01.075681 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.075651    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"]
Oct 30 04:55:01.082843 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.082809    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:55:01.083003 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.082984    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:55:01.122673 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.122632    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2526b9d9-v8zp5\" (UID: \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:01.122843 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.122694    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2526b9d9-v8zp5\" (UID: \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:01.122843 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.122793    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjjjd\" (UniqueName: \"kubernetes.io/projected/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-kube-api-access-vjjjd\") pod \"maas-api-revoke-sub-2526b9d9-v8zp5\" (UID: \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:01.223753 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.223713    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2526b9d9-v8zp5\" (UID: \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:01.223924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.223777    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2526b9d9-v8zp5\" (UID: \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:01.223924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.223809    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjjjd\" (UniqueName: \"kubernetes.io/projected/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-kube-api-access-vjjjd\") pod \"maas-api-revoke-sub-2526b9d9-v8zp5\" (UID: \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:01.224574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.224550    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2526b9d9-v8zp5\" (UID: \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:01.226409 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.226390    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2526b9d9-v8zp5\" (UID: \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:01.248309 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.248282    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjjjd\" (UniqueName: \"kubernetes.io/projected/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-kube-api-access-vjjjd\") pod \"maas-api-revoke-sub-2526b9d9-v8zp5\" (UID: \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:01.375816 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.375733    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:01.684010 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:55:01.683975    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104bfbf8_92e2_4dab_82a7_7b2ff1771d53.slice/crio-3edb603611adf284d741dcc942add58f09192a67566ce0136f154fd24da4fe6d WatchSource:0}: Error finding container 3edb603611adf284d741dcc942add58f09192a67566ce0136f154fd24da4fe6d: Status 404 returned error can't find the container with id 3edb603611adf284d741dcc942add58f09192a67566ce0136f154fd24da4fe6d
Oct 30 04:55:01.685491 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.685462    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"]
Oct 30 04:55:01.969732 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:01.969696    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5" event={"ID":"104bfbf8-92e2-4dab-82a7-7b2ff1771d53","Type":"ContainerStarted","Data":"3edb603611adf284d741dcc942add58f09192a67566ce0136f154fd24da4fe6d"}
Oct 30 04:55:02.980091 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:02.980044    5072 generic.go:356] "Generic (PLEG): container finished" podID="104bfbf8-92e2-4dab-82a7-7b2ff1771d53" containerID="ff3a1ffd1d6e03b9de18ec9deb3581355f370a0fb2d3f72759114f8b5af6b82d" exitCode=0
Oct 30 04:55:02.980548 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:02.980136    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5" event={"ID":"104bfbf8-92e2-4dab-82a7-7b2ff1771d53","Type":"ContainerDied","Data":"ff3a1ffd1d6e03b9de18ec9deb3581355f370a0fb2d3f72759114f8b5af6b82d"}
Oct 30 04:55:04.163812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:04.163783    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:04.272509 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:04.272422    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-kube-api-access-vjjjd\" (UniqueName: \"kubernetes.io/projected/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-kube-api-access-vjjjd\") pod \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\" (UID: \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\") "
Oct 30 04:55:04.272509 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:04.272485    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-xks-ca\") pod \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\" (UID: \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\") "
Oct 30 04:55:04.272739 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:04.272713    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-openshift-ca\") pod \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\" (UID: \"104bfbf8-92e2-4dab-82a7-7b2ff1771d53\") "
Oct 30 04:55:04.273025 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:04.272998    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-openshift-ca" pod "104bfbf8-92e2-4dab-82a7-7b2ff1771d53" (UID: "104bfbf8-92e2-4dab-82a7-7b2ff1771d53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:55:04.273300 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:04.273280    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:04.274815 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:04.274792    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-xks-ca" pod "104bfbf8-92e2-4dab-82a7-7b2ff1771d53" (UID: "104bfbf8-92e2-4dab-82a7-7b2ff1771d53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:55:04.274909 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:04.274889    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-kube-api-access-vjjjd" pod "104bfbf8-92e2-4dab-82a7-7b2ff1771d53" (UID: "104bfbf8-92e2-4dab-82a7-7b2ff1771d53"). InnerVolumeSpecName "kube-api-access-vjjjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:55:04.374288 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:04.374266    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjjjd\" (UniqueName: \"kubernetes.io/projected/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-kube-api-access-vjjjd\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:04.374288 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:04.374288    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104bfbf8-92e2-4dab-82a7-7b2ff1771d53-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:04.904102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:04.904059    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:55:05.000505 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:05.000458    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5" event={"ID":"104bfbf8-92e2-4dab-82a7-7b2ff1771d53","Type":"ContainerDied","Data":"3edb603611adf284d741dcc942add58f09192a67566ce0136f154fd24da4fe6d"}
Oct 30 04:55:05.000505 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:05.000482    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"
Oct 30 04:55:05.000752 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:05.000521    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3edb603611adf284d741dcc942add58f09192a67566ce0136f154fd24da4fe6d"
Oct 30 04:55:05.915528 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:05.915501    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:55:05.994929 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:05.994891    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"]
Oct 30 04:55:09.956215 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:09.956187    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"
Oct 30 04:55:10.995986 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:10.995949    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq" podUID="16f15185-8dba-42d5-b4fe-ba7a06f5b7cb" containerName="payload-processing" containerID="cri-o://d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52" gracePeriod=25
Oct 30 04:55:11.282636 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.282607    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:55:11.369739 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.369701    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-provider-credentials\") pod \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") "
Oct 30 04:55:11.369956 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.369798    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-routing-overlay-volume\") pod \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") "
Oct 30 04:55:11.369956 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.369861    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-plugins-config-volume\") pod \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") "
Oct 30 04:55:11.369956 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.369945    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-kube-api-access-dpvkx\" (UniqueName: \"kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-kube-api-access-dpvkx\") pod \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\" (UID: \"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb\") "
Oct 30 04:55:11.370343 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.370312    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-routing-overlay-volume" pod "16f15185-8dba-42d5-b4fe-ba7a06f5b7cb" (UID: "16f15185-8dba-42d5-b4fe-ba7a06f5b7cb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:55:11.370490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.370387    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-plugins-config-volume" pod "16f15185-8dba-42d5-b4fe-ba7a06f5b7cb" (UID: "16f15185-8dba-42d5-b4fe-ba7a06f5b7cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:55:11.370722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.370649    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:11.370722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.370671    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:11.372424 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.372391    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-provider-credentials" pod "16f15185-8dba-42d5-b4fe-ba7a06f5b7cb" (UID: "16f15185-8dba-42d5-b4fe-ba7a06f5b7cb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:55:11.372424 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.372404    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-kube-api-access-dpvkx" pod "16f15185-8dba-42d5-b4fe-ba7a06f5b7cb" (UID: "16f15185-8dba-42d5-b4fe-ba7a06f5b7cb"). InnerVolumeSpecName "kube-api-access-dpvkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:55:11.471698 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.471665    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpvkx\" (UniqueName: \"kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-kube-api-access-dpvkx\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:11.471698 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:11.471693    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:11.756727 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:11.756695    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:55:12.082608 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:12.082526    5072 generic.go:356] "Generic (PLEG): container finished" podID="16f15185-8dba-42d5-b4fe-ba7a06f5b7cb" containerID="d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52" exitCode=0
Oct 30 04:55:12.082608 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:12.082589    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"
Oct 30 04:55:12.083256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:12.082588    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq" event={"ID":"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb","Type":"ContainerDied","Data":"d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52"}
Oct 30 04:55:12.083256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:12.082734    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq" event={"ID":"16f15185-8dba-42d5-b4fe-ba7a06f5b7cb","Type":"ContainerDied","Data":"27657c5c8c91b8f40fb489121a73ca744888f52b50b2ef815b02b2d38985ac87"}
Oct 30 04:55:12.083256 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:12.082762    5072 scope.go:122] "RemoveContainer" containerID="d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52"
Oct 30 04:55:12.102164 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:12.102114    5072 scope.go:122] "RemoveContainer" containerID="d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52"
Oct 30 04:55:12.102519 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:12.102456    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52\": container with ID starting with d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52 not found: ID does not exist" containerID="d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52"
Oct 30 04:55:12.102519 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:12.102492    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52"} err="failed to get container status \"d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52\": rpc error: code = NotFound desc = could not find container \"d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52\": container with ID starting with d9ebb9505824ae35179893b8d688e020fc244d581170747867537520ee03bf52 not found: ID does not exist"
Oct 30 04:55:12.139977 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:12.139941    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"]
Oct 30 04:55:12.152397 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:12.152370    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7s54lq"]
Oct 30 04:55:12.419783 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:12.419702    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f15185-8dba-42d5-b4fe-ba7a06f5b7cb" path="/var/lib/kubelet/pods/16f15185-8dba-42d5-b4fe-ba7a06f5b7cb/volumes"
Oct 30 04:55:13.198542 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.198514    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"]
Oct 30 04:55:13.199348 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.199332    5072 cpu_manager.go:414] "containerMap: removing container" podUID="104bfbf8-92e2-4dab-82a7-7b2ff1771d53" containerName="revoke-keys"
Oct 30 04:55:13.199417 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.199349    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="104bfbf8-92e2-4dab-82a7-7b2ff1771d53" containerName="revoke-keys"
Oct 30 04:55:13.199417 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.199395    5072 cpu_manager.go:414] "containerMap: removing container" podUID="16f15185-8dba-42d5-b4fe-ba7a06f5b7cb" containerName="payload-processing"
Oct 30 04:55:13.199417 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.199400    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f15185-8dba-42d5-b4fe-ba7a06f5b7cb" containerName="payload-processing"
Oct 30 04:55:13.199522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.199513    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f15185-8dba-42d5-b4fe-ba7a06f5b7cb" containerName="payload-processing"
Oct 30 04:55:13.199557 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.199527    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="104bfbf8-92e2-4dab-82a7-7b2ff1771d53" containerName="revoke-keys"
Oct 30 04:55:13.204766 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.204746    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.223487 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.223452    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"]
Oct 30 04:55:13.299864 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.299832    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4bhr\" (UniqueName: \"kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-kube-api-access-w4bhr\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.300172 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.300145    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.300295 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.300273    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.300420 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.300314    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.401212 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.401180    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.401391 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.401266    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.401391 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.401298    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.401391 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.401332    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bhr\" (UniqueName: \"kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-kube-api-access-w4bhr\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.401923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.401893    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.402058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.402039    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.403969 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.403948    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.426028 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.425992    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bhr\" (UniqueName: \"kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-kube-api-access-w4bhr\") pod \"payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") " pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.525638 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.525549    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:13.870177 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:55:13.870146    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d802f1b_5fa6_4453_b3d3_9c58f3d63cbf.slice/crio-80897a4cb26d0dabc0ba63dcae844bf9a0804c53a991ecc25749a6404018e320 WatchSource:0}: Error finding container 80897a4cb26d0dabc0ba63dcae844bf9a0804c53a991ecc25749a6404018e320: Status 404 returned error can't find the container with id 80897a4cb26d0dabc0ba63dcae844bf9a0804c53a991ecc25749a6404018e320
Oct 30 04:55:13.871686 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:13.871656    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"]
Oct 30 04:55:14.103994 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:14.103958    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" event={"ID":"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf","Type":"ContainerStarted","Data":"a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3"}
Oct 30 04:55:14.104209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:14.104014    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" event={"ID":"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf","Type":"ContainerStarted","Data":"80897a4cb26d0dabc0ba63dcae844bf9a0804c53a991ecc25749a6404018e320"}
Oct 30 04:55:14.104209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:14.104056    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:14.152927 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:14.152839    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" podStartSLOduration=1.1528264830000001 podStartE2EDuration="1.152826483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:14.150723516 +0000 UTC m=+3493.892146613" watchObservedRunningTime="2026-10-30 04:55:14.152826483 +0000 UTC m=+3493.894249597"
Oct 30 04:55:14.728619 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:14.728588    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"]
Oct 30 04:55:14.732740 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:14.732713    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"]
Oct 30 04:55:15.119178 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:15.119060    5072 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" secret="" err="secret \"payload-processing-external-model-e2e--e5d834c2-dockercfg-bcxjw\" not found"
Oct 30 04:55:15.228317 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:15.228280    5072 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-plugins-e2e-extmod-a-c0e964: configmap "payload-processing-external-model-plugins-e2e-extmod-a-c0e964" not found
Oct 30 04:55:15.228513 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:15.228330    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c0e964/agc-cred-d17fe97dadf9: secret "agc-cred-d17fe97dadf9" not found
Oct 30 04:55:15.228513 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:15.228362    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4: secret "agc-cred-d17fe97dadf9" not found
Oct 30 04:55:15.228513 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:15.228374    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume podName:1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf nodeName:}" failed. No retries permitted until 2026-10-30 04:55:15.728358197 +0000 UTC m=+3495.469781271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" (UID: "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-c0e964" not found
Oct 30 04:55:15.228513 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:15.228405    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials podName:1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf nodeName:}" failed. No retries permitted until 2026-10-30 04:55:15.728392298 +0000 UTC m=+3495.469815399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" (UID: "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf") : secret "agc-cred-d17fe97dadf9" not found
Oct 30 04:55:15.733728 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:15.733700    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c0e964/agc-cred-d17fe97dadf9: secret "agc-cred-d17fe97dadf9" not found
Oct 30 04:55:15.733728 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:15.733719    5072 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-plugins-e2e-extmod-a-c0e964: configmap "payload-processing-external-model-plugins-e2e-extmod-a-c0e964" not found
Oct 30 04:55:15.734183 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:15.733812    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume podName:1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf nodeName:}" failed. No retries permitted until 2026-10-30 04:55:16.733792519 +0000 UTC m=+3496.475215608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" (UID: "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-c0e964" not found
Oct 30 04:55:15.734183 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:15.733726    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4: secret "agc-cred-d17fe97dadf9" not found
Oct 30 04:55:15.734183 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:15.733858    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials podName:1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf nodeName:}" failed. No retries permitted until 2026-10-30 04:55:16.73384709 +0000 UTC m=+3496.475270165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" (UID: "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf") : secret "agc-cred-d17fe97dadf9" not found
Oct 30 04:55:16.040412 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:16.037903    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"]
Oct 30 04:55:16.051411 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:16.051378    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29de31b-m54hr"]
Oct 30 04:55:16.411070 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:16.410996    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe57e396-3a5b-491e-bc59-b70fd9487ae5" path="/var/lib/kubelet/pods/fe57e396-3a5b-491e-bc59-b70fd9487ae5/volumes"
Oct 30 04:55:16.752295 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:16.752254    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c0e964/agc-cred-d17fe97dadf9: secret "agc-cred-d17fe97dadf9" not found
Oct 30 04:55:16.752295 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:16.752293    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4: secret "agc-cred-d17fe97dadf9" not found
Oct 30 04:55:16.752735 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:16.752337    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials podName:1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf nodeName:}" failed. No retries permitted until 2026-10-30 04:55:18.752323603 +0000 UTC m=+3498.493746691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" (UID: "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf") : secret "agc-cred-d17fe97dadf9" not found
Oct 30 04:55:16.752735 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:16.752270    5072 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-plugins-e2e-extmod-a-c0e964: configmap "payload-processing-external-model-plugins-e2e-extmod-a-c0e964" not found
Oct 30 04:55:16.752735 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:16.752426    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume podName:1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf nodeName:}" failed. No retries permitted until 2026-10-30 04:55:18.752412467 +0000 UTC m=+3498.493835558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" (UID: "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-c0e964" not found
Oct 30 04:55:18.799311 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:18.799272    5072 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-plugins-e2e-extmod-a-c0e964: configmap "payload-processing-external-model-plugins-e2e-extmod-a-c0e964" not found
Oct 30 04:55:18.799742 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:18.799351    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c0e964/agc-cred-d17fe97dadf9: secret "agc-cred-d17fe97dadf9" not found
Oct 30 04:55:18.799742 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:18.799367    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4: secret "agc-cred-d17fe97dadf9" not found
Oct 30 04:55:18.799742 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:18.799357    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume podName:1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf nodeName:}" failed. No retries permitted until 2026-10-30 04:55:22.799339287 +0000 UTC m=+3502.540762380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" (UID: "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-c0e964" not found
Oct 30 04:55:18.799742 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:18.799411    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials podName:1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf nodeName:}" failed. No retries permitted until 2026-10-30 04:55:22.799400234 +0000 UTC m=+3502.540823311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" (UID: "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf") : secret "agc-cred-d17fe97dadf9" not found
Oct 30 04:55:19.728858 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:19.728822    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms" podUID="1c8c0ab0-a99a-4116-b0fb-e381049a5cbf" containerName="payload-processing" containerID="cri-o://da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150" gracePeriod=25
Oct 30 04:55:20.014818 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.014787    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:55:20.121703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.121662    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-routing-overlay-volume\") pod \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") "
Oct 30 04:55:20.121895 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.121790    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-provider-credentials\") pod \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") "
Oct 30 04:55:20.122006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.121992    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-plugins-config-volume\") pod \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") "
Oct 30 04:55:20.122072 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.122000    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-routing-overlay-volume" pod "1c8c0ab0-a99a-4116-b0fb-e381049a5cbf" (UID: "1c8c0ab0-a99a-4116-b0fb-e381049a5cbf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:55:20.122163 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.122090    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-kube-api-access-xbtcv\" (UniqueName: \"kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-kube-api-access-xbtcv\") pod \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\" (UID: \"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf\") "
Oct 30 04:55:20.122518 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.122482    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-plugins-config-volume" pod "1c8c0ab0-a99a-4116-b0fb-e381049a5cbf" (UID: "1c8c0ab0-a99a-4116-b0fb-e381049a5cbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:55:20.122728 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.122708    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:20.122795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.122734    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:20.124207 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.124182    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-provider-credentials" pod "1c8c0ab0-a99a-4116-b0fb-e381049a5cbf" (UID: "1c8c0ab0-a99a-4116-b0fb-e381049a5cbf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:55:20.124625 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.124606    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-kube-api-access-xbtcv" pod "1c8c0ab0-a99a-4116-b0fb-e381049a5cbf" (UID: "1c8c0ab0-a99a-4116-b0fb-e381049a5cbf"). InnerVolumeSpecName "kube-api-access-xbtcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:55:20.183176 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.183147    5072 generic.go:356] "Generic (PLEG): container finished" podID="1c8c0ab0-a99a-4116-b0fb-e381049a5cbf" containerID="da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150" exitCode=0
Oct 30 04:55:20.183271 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.183185    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"
Oct 30 04:55:20.183271 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.183223    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms" event={"ID":"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf","Type":"ContainerDied","Data":"da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150"}
Oct 30 04:55:20.183346 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.183278    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms" event={"ID":"1c8c0ab0-a99a-4116-b0fb-e381049a5cbf","Type":"ContainerDied","Data":"1db2bbd0e6afe3245d1eb52ef15cf9d73e8d0e8ca46e3e51aad02437b69561b6"}
Oct 30 04:55:20.183346 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.183302    5072 scope.go:122] "RemoveContainer" containerID="da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150"
Oct 30 04:55:20.195290 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.195273    5072 scope.go:122] "RemoveContainer" containerID="da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150"
Oct 30 04:55:20.195554 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:20.195536    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150\": container with ID starting with da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150 not found: ID does not exist" containerID="da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150"
Oct 30 04:55:20.195597 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.195561    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150"} err="failed to get container status \"da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150\": rpc error: code = NotFound desc = could not find container \"da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150\": container with ID starting with da84a0579d51f60fd3934b57c85b6ee8bd3a0942eb71e7349d825c37a7514150 not found: ID does not exist"
Oct 30 04:55:20.223873 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.223849    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbtcv\" (UniqueName: \"kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-kube-api-access-xbtcv\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:20.223873 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.223872    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:20.238973 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.238918    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"]
Oct 30 04:55:20.244532 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.244507    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-59bfq5vms"]
Oct 30 04:55:20.409634 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:20.409596    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8c0ab0-a99a-4116-b0fb-e381049a5cbf" path="/var/lib/kubelet/pods/1c8c0ab0-a99a-4116-b0fb-e381049a5cbf/volumes"
Oct 30 04:55:21.150109 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.150030    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" podUID="1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf" containerName="payload-processing" containerID="cri-o://a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3" gracePeriod=25
Oct 30 04:55:21.456819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.456788    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:21.543140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.543104    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-routing-overlay-volume\") pod \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") "
Oct 30 04:55:21.543361 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.543240    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials\") pod \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") "
Oct 30 04:55:21.543361 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.543342    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-kube-api-access-w4bhr\" (UniqueName: \"kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-kube-api-access-w4bhr\") pod \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") "
Oct 30 04:55:21.543520 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.543497    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume\") pod \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\" (UID: \"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf\") "
Oct 30 04:55:21.543601 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.543520    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-routing-overlay-volume" pod "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf" (UID: "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:55:21.544444 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.543835    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume" pod "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf" (UID: "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:55:21.544588 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.544564    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:21.544656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.544590    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:21.546240 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.546206    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials" pod "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf" (UID: "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:55:21.546591 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.546568    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-kube-api-access-w4bhr" pod "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf" (UID: "1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf"). InnerVolumeSpecName "kube-api-access-w4bhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:55:21.645869 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.645841    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:21.645869 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:21.645869    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4bhr\" (UniqueName: \"kubernetes.io/projected/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf-kube-api-access-w4bhr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:22.201621 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:22.201584    5072 generic.go:356] "Generic (PLEG): container finished" podID="1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf" containerID="a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3" exitCode=0
Oct 30 04:55:22.202044 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:22.201667    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"
Oct 30 04:55:22.202044 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:22.201672    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" event={"ID":"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf","Type":"ContainerDied","Data":"a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3"}
Oct 30 04:55:22.202044 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:22.201854    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4" event={"ID":"1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf","Type":"ContainerDied","Data":"80897a4cb26d0dabc0ba63dcae844bf9a0804c53a991ecc25749a6404018e320"}
Oct 30 04:55:22.202044 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:22.201882    5072 scope.go:122] "RemoveContainer" containerID="a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3"
Oct 30 04:55:22.214560 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:22.214537    5072 scope.go:122] "RemoveContainer" containerID="a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3"
Oct 30 04:55:22.214831 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:22.214804    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3\": container with ID starting with a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3 not found: ID does not exist" containerID="a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3"
Oct 30 04:55:22.214934 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:22.214840    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3"} err="failed to get container status \"a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3\": rpc error: code = NotFound desc = could not find container \"a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3\": container with ID starting with a6777429ffb4d5e8f4063fa59a9876b76d3dd013cba8f2ee6deb230d0418e0d3 not found: ID does not exist"
Oct 30 04:55:22.270134 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:22.270101    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"]
Oct 30 04:55:22.281102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:22.279790    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c0e964/payload-processing-external-model-e2e-extmod-a-c0e964-ff8647vx4"]
Oct 30 04:55:22.419627 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:22.419596    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf" path="/var/lib/kubelet/pods/1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf/volumes"
Oct 30 04:55:28.035669 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.035624    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"]
Oct 30 04:55:28.050272 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.050241    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c45f5ab0-rqrwj"]
Oct 30 04:55:28.411703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.411628    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b12704-1363-4bec-867f-e3923d9b0fad" path="/var/lib/kubelet/pods/f1b12704-1363-4bec-867f-e3923d9b0fad/volumes"
Oct 30 04:55:28.665034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.664947    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"]
Oct 30 04:55:28.665851 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.665834    5072 cpu_manager.go:414] "containerMap: removing container" podUID="1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf" containerName="payload-processing"
Oct 30 04:55:28.665912 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.665853    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf" containerName="payload-processing"
Oct 30 04:55:28.665912 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.665872    5072 cpu_manager.go:414] "containerMap: removing container" podUID="1c8c0ab0-a99a-4116-b0fb-e381049a5cbf" containerName="payload-processing"
Oct 30 04:55:28.665912 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.665877    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8c0ab0-a99a-4116-b0fb-e381049a5cbf" containerName="payload-processing"
Oct 30 04:55:28.666036 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.665983    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8c0ab0-a99a-4116-b0fb-e381049a5cbf" containerName="payload-processing"
Oct 30 04:55:28.666036 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.666002    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d802f1b-5fa6-4453-b3d3-9c58f3d63cbf" containerName="payload-processing"
Oct 30 04:55:28.671343 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.671317    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"
Oct 30 04:55:28.689627 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.689603    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"]
Oct 30 04:55:28.697264 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.697243    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-449d83\""
Oct 30 04:55:28.725747 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.725718    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"]
Oct 30 04:55:28.730539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.730523    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"
Oct 30 04:55:28.746167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.746125    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-449d83-dockercfg-p9v9c\""
Oct 30 04:55:28.746950 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.746159    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dc2b2c-76e1-4433-962c-179a352a8e8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5\" (UID: \"e9dc2b2c-76e1-4433-962c-179a352a8e8f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"
Oct 30 04:55:28.746950 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.746415    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvz79\" (UniqueName: \"kubernetes.io/projected/e9dc2b2c-76e1-4433-962c-179a352a8e8f-kube-api-access-rvz79\") pod \"payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5\" (UID: \"e9dc2b2c-76e1-4433-962c-179a352a8e8f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"
Oct 30 04:55:28.750010 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.749991    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"]
Oct 30 04:55:28.848646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.848610    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a562af18-3e2a-493a-838d-0966f513c9c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7\" (UID: \"a562af18-3e2a-493a-838d-0966f513c9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"
Oct 30 04:55:28.848922 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.848702    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvz79\" (UniqueName: \"kubernetes.io/projected/e9dc2b2c-76e1-4433-962c-179a352a8e8f-kube-api-access-rvz79\") pod \"payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5\" (UID: \"e9dc2b2c-76e1-4433-962c-179a352a8e8f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"
Oct 30 04:55:28.848922 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.848850    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzgh\" (UniqueName: \"kubernetes.io/projected/a562af18-3e2a-493a-838d-0966f513c9c2-kube-api-access-qjzgh\") pod \"payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7\" (UID: \"a562af18-3e2a-493a-838d-0966f513c9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"
Oct 30 04:55:28.848922 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.848884    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dc2b2c-76e1-4433-962c-179a352a8e8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5\" (UID: \"e9dc2b2c-76e1-4433-962c-179a352a8e8f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"
Oct 30 04:55:28.850042 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.850004    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dc2b2c-76e1-4433-962c-179a352a8e8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5\" (UID: \"e9dc2b2c-76e1-4433-962c-179a352a8e8f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"
Oct 30 04:55:28.884660 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.884627    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvz79\" (UniqueName: \"kubernetes.io/projected/e9dc2b2c-76e1-4433-962c-179a352a8e8f-kube-api-access-rvz79\") pod \"payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5\" (UID: \"e9dc2b2c-76e1-4433-962c-179a352a8e8f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"
Oct 30 04:55:28.949748 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.949710    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a562af18-3e2a-493a-838d-0966f513c9c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7\" (UID: \"a562af18-3e2a-493a-838d-0966f513c9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"
Oct 30 04:55:28.949939 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.949849    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzgh\" (UniqueName: \"kubernetes.io/projected/a562af18-3e2a-493a-838d-0966f513c9c2-kube-api-access-qjzgh\") pod \"payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7\" (UID: \"a562af18-3e2a-493a-838d-0966f513c9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"
Oct 30 04:55:28.950517 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.950493    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a562af18-3e2a-493a-838d-0966f513c9c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7\" (UID: \"a562af18-3e2a-493a-838d-0966f513c9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"
Oct 30 04:55:28.974404 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.974383    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzgh\" (UniqueName: \"kubernetes.io/projected/a562af18-3e2a-493a-838d-0966f513c9c2-kube-api-access-qjzgh\") pod \"payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7\" (UID: \"a562af18-3e2a-493a-838d-0966f513c9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"
Oct 30 04:55:28.987572 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:28.987551    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"
Oct 30 04:55:29.048790 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:29.048524    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"
Oct 30 04:55:29.296734 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:55:29.296703    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9dc2b2c_76e1_4433_962c_179a352a8e8f.slice/crio-2727b7c02d63f9549b5f048e6406c9e93aa58bb8033795a5f77bdbfe7cbdbc06 WatchSource:0}: Error finding container 2727b7c02d63f9549b5f048e6406c9e93aa58bb8033795a5f77bdbfe7cbdbc06: Status 404 returned error can't find the container with id 2727b7c02d63f9549b5f048e6406c9e93aa58bb8033795a5f77bdbfe7cbdbc06
Oct 30 04:55:29.297164 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:29.297139    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"]
Oct 30 04:55:29.372937 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:55:29.372906    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda562af18_3e2a_493a_838d_0966f513c9c2.slice/crio-2a7dabf3cf7eeb41ccc761cf1ec0e816e962ad49d9645e0e8a06c4a479e27083 WatchSource:0}: Error finding container 2a7dabf3cf7eeb41ccc761cf1ec0e816e962ad49d9645e0e8a06c4a479e27083: Status 404 returned error can't find the container with id 2a7dabf3cf7eeb41ccc761cf1ec0e816e962ad49d9645e0e8a06c4a479e27083
Oct 30 04:55:29.374317 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:29.374282    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"]
Oct 30 04:55:30.283723 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:30.283680    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5" event={"ID":"e9dc2b2c-76e1-4433-962c-179a352a8e8f","Type":"ContainerStarted","Data":"cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541"}
Oct 30 04:55:30.284408 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:30.283748    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5" event={"ID":"e9dc2b2c-76e1-4433-962c-179a352a8e8f","Type":"ContainerStarted","Data":"2727b7c02d63f9549b5f048e6406c9e93aa58bb8033795a5f77bdbfe7cbdbc06"}
Oct 30 04:55:30.284796 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:30.284771    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"
Oct 30 04:55:30.286971 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:30.286942    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7" event={"ID":"a562af18-3e2a-493a-838d-0966f513c9c2","Type":"ContainerStarted","Data":"8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df"}
Oct 30 04:55:30.287232 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:30.286997    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7" event={"ID":"a562af18-3e2a-493a-838d-0966f513c9c2","Type":"ContainerStarted","Data":"2a7dabf3cf7eeb41ccc761cf1ec0e816e962ad49d9645e0e8a06c4a479e27083"}
Oct 30 04:55:30.334316 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:30.334252    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5" podStartSLOduration=2.33423376 podStartE2EDuration="2.33423376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:30.333973423 +0000 UTC m=+3510.075396519" watchObservedRunningTime="2026-10-30 04:55:30.33423376 +0000 UTC m=+3510.075656858"
Oct 30 04:55:30.376038 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:30.375989    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7" podStartSLOduration=2.375974852 podStartE2EDuration="2.375974852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:30.37496525 +0000 UTC m=+3510.116388347" watchObservedRunningTime="2026-10-30 04:55:30.375974852 +0000 UTC m=+3510.117398372"
Oct 30 04:55:31.295130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:31.295100    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"
Oct 30 04:55:40.031593 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:40.031547    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"]
Oct 30 04:55:40.043819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:40.043789    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-713164b5-5wxlk"]
Oct 30 04:55:40.409747 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:40.409673    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d68385-595f-4d22-b761-e2be6ce315e4" path="/var/lib/kubelet/pods/f5d68385-595f-4d22-b761-e2be6ce315e4/volumes"
Oct 30 04:55:41.757690 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:55:41.757650    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:55:42.306890 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:42.306853    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"
Oct 30 04:55:42.307070 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:42.306933    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"
Oct 30 04:55:49.747631 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.747597    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"]
Oct 30 04:55:49.752469 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.752452    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:49.787263 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.787229    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"]
Oct 30 04:55:49.791129 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.789689    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:55:49.791129 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.789760    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:55:49.851731 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.851697    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0421600d-dn5tg\" (UID: \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:49.851891 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.851816    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjbn\" (UniqueName: \"kubernetes.io/projected/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-kube-api-access-rvjbn\") pod \"maas-api-revoke-sub-0421600d-dn5tg\" (UID: \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:49.852005 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.851983    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0421600d-dn5tg\" (UID: \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:49.952955 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.952923    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0421600d-dn5tg\" (UID: \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:49.953179 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.953008    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjbn\" (UniqueName: \"kubernetes.io/projected/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-kube-api-access-rvjbn\") pod \"maas-api-revoke-sub-0421600d-dn5tg\" (UID: \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:49.953179 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.953163    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0421600d-dn5tg\" (UID: \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:49.953894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.953867    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0421600d-dn5tg\" (UID: \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:49.955841 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.955814    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0421600d-dn5tg\" (UID: \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:49.986895 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:49.986869    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjbn\" (UniqueName: \"kubernetes.io/projected/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-kube-api-access-rvjbn\") pod \"maas-api-revoke-sub-0421600d-dn5tg\" (UID: \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:50.098649 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:50.098561    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:50.433633 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:55:50.433596    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddddc8322_0daf_4ea0_bce8_9fba2f71e0e2.slice/crio-5540a46c228f8778e76d8f68f5c0988e6a24eaf213d7f89b0e7840cc0c66a539 WatchSource:0}: Error finding container 5540a46c228f8778e76d8f68f5c0988e6a24eaf213d7f89b0e7840cc0c66a539: Status 404 returned error can't find the container with id 5540a46c228f8778e76d8f68f5c0988e6a24eaf213d7f89b0e7840cc0c66a539
Oct 30 04:55:50.434560 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:50.434535    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"]
Oct 30 04:55:50.495122 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:50.493923    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg" event={"ID":"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2","Type":"ContainerStarted","Data":"5540a46c228f8778e76d8f68f5c0988e6a24eaf213d7f89b0e7840cc0c66a539"}
Oct 30 04:55:51.504647 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:51.504603    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg" event={"ID":"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2","Type":"ContainerStarted","Data":"5dc8845bdd13a3e58633f62b081342d22ace08b7941d3eb76cb6deebf0fe0b8c"}
Oct 30 04:55:51.546434 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:51.546380    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg" podStartSLOduration=2.546366143 podStartE2EDuration="2.546366143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:51.545929407 +0000 UTC m=+3531.287352528" watchObservedRunningTime="2026-10-30 04:55:51.546366143 +0000 UTC m=+3531.287789240"
Oct 30 04:55:52.519167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:52.519128    5072 generic.go:356] "Generic (PLEG): container finished" podID="dddc8322-0daf-4ea0-bce8-9fba2f71e0e2" containerID="5dc8845bdd13a3e58633f62b081342d22ace08b7941d3eb76cb6deebf0fe0b8c" exitCode=0
Oct 30 04:55:52.519608 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:52.519207    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg" event={"ID":"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2","Type":"ContainerDied","Data":"5dc8845bdd13a3e58633f62b081342d22ace08b7941d3eb76cb6deebf0fe0b8c"}
Oct 30 04:55:53.716673 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:53.716639    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:53.834632 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:53.834548    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-kube-api-access-rvjbn\" (UniqueName: \"kubernetes.io/projected/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-kube-api-access-rvjbn\") pod \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\" (UID: \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\") "
Oct 30 04:55:53.834790 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:53.834684    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-xks-ca\") pod \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\" (UID: \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\") "
Oct 30 04:55:53.834842 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:53.834796    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-openshift-ca\") pod \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\" (UID: \"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2\") "
Oct 30 04:55:53.835266 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:53.835235    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-openshift-ca" pod "dddc8322-0daf-4ea0-bce8-9fba2f71e0e2" (UID: "dddc8322-0daf-4ea0-bce8-9fba2f71e0e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:55:53.835407 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:53.835340    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:53.836903 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:53.836873    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-kube-api-access-rvjbn" pod "dddc8322-0daf-4ea0-bce8-9fba2f71e0e2" (UID: "dddc8322-0daf-4ea0-bce8-9fba2f71e0e2"). InnerVolumeSpecName "kube-api-access-rvjbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:55:53.837434 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:53.837410    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-xks-ca" pod "dddc8322-0daf-4ea0-bce8-9fba2f71e0e2" (UID: "dddc8322-0daf-4ea0-bce8-9fba2f71e0e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:55:53.936734 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:53.936691    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:53.936734 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:53.936733    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjbn\" (UniqueName: \"kubernetes.io/projected/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2-kube-api-access-rvjbn\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:55:54.539151 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:54.539115    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"
Oct 30 04:55:54.539151 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:54.539121    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg" event={"ID":"dddc8322-0daf-4ea0-bce8-9fba2f71e0e2","Type":"ContainerDied","Data":"5540a46c228f8778e76d8f68f5c0988e6a24eaf213d7f89b0e7840cc0c66a539"}
Oct 30 04:55:54.539367 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:54.539183    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5540a46c228f8778e76d8f68f5c0988e6a24eaf213d7f89b0e7840cc0c66a539"
Oct 30 04:55:56.559013 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:56.558978    5072 generic.go:356] "Generic (PLEG): container finished" podID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerID="6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd" exitCode=0
Oct 30 04:55:56.559507 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:56.559047    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" event={"ID":"bfa3c63a-eaf0-4aec-b463-adc66812e632","Type":"ContainerDied","Data":"6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd"}
Oct 30 04:55:56.559507 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:56.559482    5072 scope.go:122] "RemoveContainer" containerID="6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd"
Oct 30 04:55:56.561375 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:56.561344    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:55:56.561493 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:56.561431    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:55:56.561493 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:56.561482    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:55:57.588938 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:57.588902    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" event={"ID":"bfa3c63a-eaf0-4aec-b463-adc66812e632","Type":"ContainerStarted","Data":"0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112"}
Oct 30 04:55:58.550244 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:58.550203    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:55:58.550603 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:58.550569    5072 patch_prober.go:29] interesting pod/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 04:55:58.550728 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:58.550641    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 04:55:59.550512 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:59.550472    5072 patch_prober.go:29] interesting pod/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 04:55:59.551025 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:55:59.550551    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 04:56:00.488165 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.488125    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"]
Oct 30 04:56:00.489096 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.489059    5072 cpu_manager.go:414] "containerMap: removing container" podUID="dddc8322-0daf-4ea0-bce8-9fba2f71e0e2" containerName="revoke-keys"
Oct 30 04:56:00.489096 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.489097    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddc8322-0daf-4ea0-bce8-9fba2f71e0e2" containerName="revoke-keys"
Oct 30 04:56:00.489429 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.489410    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddc8322-0daf-4ea0-bce8-9fba2f71e0e2" containerName="revoke-keys"
Oct 30 04:56:00.494719 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.494697    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:00.510222 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.510179    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"]
Oct 30 04:56:00.515202 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.515172    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:56:00.515385 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.515192    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:56:00.550690 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.550652    5072 patch_prober.go:29] interesting pod/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 04:56:00.551236 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.550719    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 04:56:00.632640 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.632603    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcd7n\" (UniqueName: \"kubernetes.io/projected/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-kube-api-access-xcd7n\") pod \"maas-api-revoke-sub-9ebb0e2a-5vz7d\" (UID: \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:00.632839 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.632656    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ebb0e2a-5vz7d\" (UID: \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:00.632884 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.632866    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ebb0e2a-5vz7d\" (UID: \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:00.734319 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.734281    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ebb0e2a-5vz7d\" (UID: \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:00.734539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.734380    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcd7n\" (UniqueName: \"kubernetes.io/projected/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-kube-api-access-xcd7n\") pod \"maas-api-revoke-sub-9ebb0e2a-5vz7d\" (UID: \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:00.734539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.734405    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ebb0e2a-5vz7d\" (UID: \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:00.735209 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.735187    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ebb0e2a-5vz7d\" (UID: \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:00.736965 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.736942    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ebb0e2a-5vz7d\" (UID: \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:00.781893 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.781797    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcd7n\" (UniqueName: \"kubernetes.io/projected/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-kube-api-access-xcd7n\") pod \"maas-api-revoke-sub-9ebb0e2a-5vz7d\" (UID: \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:00.812855 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:00.812822    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:01.143567 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:01.143526    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"]
Oct 30 04:56:01.150751 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:56:01.150711    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3b2141_fe42_46f3_bbba_6dfcfd588e1e.slice/crio-8ff201ff6f15b3e91db582f33e29b1422267a2464e82a3fa51aeb02b501f6bc4 WatchSource:0}: Error finding container 8ff201ff6f15b3e91db582f33e29b1422267a2464e82a3fa51aeb02b501f6bc4: Status 404 returned error can't find the container with id 8ff201ff6f15b3e91db582f33e29b1422267a2464e82a3fa51aeb02b501f6bc4
Oct 30 04:56:01.551049 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:01.551013    5072 patch_prober.go:29] interesting pod/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 04:56:01.551484 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:01.551067    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 04:56:01.623951 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:01.623911    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d" event={"ID":"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e","Type":"ContainerStarted","Data":"8ff201ff6f15b3e91db582f33e29b1422267a2464e82a3fa51aeb02b501f6bc4"}
Oct 30 04:56:01.904433 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:01.904390    5072 scope.go:122] "RemoveContainer" containerID="86e87ff61c02c8e5d1b3e572b947172edd187379ca62400712d4aa489d090f90"
Oct 30 04:56:01.916686 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:01.916658    5072 scope.go:122] "RemoveContainer" containerID="a5d888638b2ceaaf7f5a09bcd51e7e073c8af8f26e34f33931c332f3108199ba"
Oct 30 04:56:01.927567 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:01.927542    5072 scope.go:122] "RemoveContainer" containerID="07efd1c81ee8bb64bdce22babf62ad630eaf81d5ead4ebb3759ccaa593705708"
Oct 30 04:56:01.938738 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:01.938711    5072 scope.go:122] "RemoveContainer" containerID="70dfedc29bfe8ad6d5343d8da31d9340ab9a3fa98321f8223270be89d2c8d064"
Oct 30 04:56:02.550512 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:02.550475    5072 patch_prober.go:29] interesting pod/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 30 04:56:02.550688 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:02.550531    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 30 04:56:02.633763 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:02.633729    5072 generic.go:356] "Generic (PLEG): container finished" podID="1b3b2141-fe42-46f3-bbba-6dfcfd588e1e" containerID="1117e6b6a49a81a73d8e259857400e016c5bb596bba62e6cf3da94b63da32c97" exitCode=0
Oct 30 04:56:02.634211 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:02.633806    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d" event={"ID":"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e","Type":"ContainerDied","Data":"1117e6b6a49a81a73d8e259857400e016c5bb596bba62e6cf3da94b63da32c97"}
Oct 30 04:56:03.555349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:03.555313    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:56:03.555814 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:03.555799    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:56:03.556608 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:03.556587    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 04:56:03.844885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:03.844860    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:03.907303 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:03.907271    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-openshift-ca\") pod \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\" (UID: \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\") "
Oct 30 04:56:03.907567 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:03.907325    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-xks-ca\") pod \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\" (UID: \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\") "
Oct 30 04:56:03.907567 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:03.907401    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-kube-api-access-xcd7n\" (UniqueName: \"kubernetes.io/projected/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-kube-api-access-xcd7n\") pod \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\" (UID: \"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e\") "
Oct 30 04:56:03.907739 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:03.907715    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-openshift-ca" pod "1b3b2141-fe42-46f3-bbba-6dfcfd588e1e" (UID: "1b3b2141-fe42-46f3-bbba-6dfcfd588e1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:03.908349 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:03.908324    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:03.910026 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:03.909999    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-xks-ca" pod "1b3b2141-fe42-46f3-bbba-6dfcfd588e1e" (UID: "1b3b2141-fe42-46f3-bbba-6dfcfd588e1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:56:03.910142 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:03.910022    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-kube-api-access-xcd7n" pod "1b3b2141-fe42-46f3-bbba-6dfcfd588e1e" (UID: "1b3b2141-fe42-46f3-bbba-6dfcfd588e1e"). InnerVolumeSpecName "kube-api-access-xcd7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:04.010147 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:04.010070    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:04.010147 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:04.010147    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcd7n\" (UniqueName: \"kubernetes.io/projected/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e-kube-api-access-xcd7n\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:04.651050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:04.651017    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"
Oct 30 04:56:04.651050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:04.651042    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d" event={"ID":"1b3b2141-fe42-46f3-bbba-6dfcfd588e1e","Type":"ContainerDied","Data":"8ff201ff6f15b3e91db582f33e29b1422267a2464e82a3fa51aeb02b501f6bc4"}
Oct 30 04:56:04.651280 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:04.651107    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ff201ff6f15b3e91db582f33e29b1422267a2464e82a3fa51aeb02b501f6bc4"
Oct 30 04:56:08.030955 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:08.030920    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"]
Oct 30 04:56:08.043449 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:08.043420    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b3aaa1-fxcmp"]
Oct 30 04:56:08.410947 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:08.410868    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc076262-cc82-4d9a-a09e-f3f789ca6e6c" path="/var/lib/kubelet/pods/fc076262-cc82-4d9a-a09e-f3f789ca6e6c/volumes"
Oct 30 04:56:10.520972 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.520896    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"]
Oct 30 04:56:10.521823 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.521797    5072 cpu_manager.go:414] "containerMap: removing container" podUID="1b3b2141-fe42-46f3-bbba-6dfcfd588e1e" containerName="revoke-keys"
Oct 30 04:56:10.521823 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.521818    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3b2141-fe42-46f3-bbba-6dfcfd588e1e" containerName="revoke-keys"
Oct 30 04:56:10.521972 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.521960    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3b2141-fe42-46f3-bbba-6dfcfd588e1e" containerName="revoke-keys"
Oct 30 04:56:10.526167 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.526149    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:10.544916 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.544888    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"]
Oct 30 04:56:10.552237 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.552208    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:56:10.552376 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.552211    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:56:10.615928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.615898    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77ea49f6-sw4m2\" (UID: \"0e57e6b3-15af-4562-87de-48093607a0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:10.616118 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.616038    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk4tx\" (UniqueName: \"kubernetes.io/projected/0e57e6b3-15af-4562-87de-48093607a0f4-kube-api-access-nk4tx\") pod \"maas-api-revoke-sub-77ea49f6-sw4m2\" (UID: \"0e57e6b3-15af-4562-87de-48093607a0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:10.616178 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.616151    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77ea49f6-sw4m2\" (UID: \"0e57e6b3-15af-4562-87de-48093607a0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:10.717797 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.717748    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk4tx\" (UniqueName: \"kubernetes.io/projected/0e57e6b3-15af-4562-87de-48093607a0f4-kube-api-access-nk4tx\") pod \"maas-api-revoke-sub-77ea49f6-sw4m2\" (UID: \"0e57e6b3-15af-4562-87de-48093607a0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:10.718132 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.717806    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77ea49f6-sw4m2\" (UID: \"0e57e6b3-15af-4562-87de-48093607a0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:10.718132 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.718012    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77ea49f6-sw4m2\" (UID: \"0e57e6b3-15af-4562-87de-48093607a0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:10.718833 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.718806    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77ea49f6-sw4m2\" (UID: \"0e57e6b3-15af-4562-87de-48093607a0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:10.720987 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.720954    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77ea49f6-sw4m2\" (UID: \"0e57e6b3-15af-4562-87de-48093607a0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:10.742539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.742510    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk4tx\" (UniqueName: \"kubernetes.io/projected/0e57e6b3-15af-4562-87de-48093607a0f4-kube-api-access-nk4tx\") pod \"maas-api-revoke-sub-77ea49f6-sw4m2\" (UID: \"0e57e6b3-15af-4562-87de-48093607a0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:10.841539 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:10.841474    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:11.147855 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:56:11.147825    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e57e6b3_15af_4562_87de_48093607a0f4.slice/crio-8f326d0056f9b4be0c1cecca5de896146195ddd6d479addb8384ec3ad39e6d11 WatchSource:0}: Error finding container 8f326d0056f9b4be0c1cecca5de896146195ddd6d479addb8384ec3ad39e6d11: Status 404 returned error can't find the container with id 8f326d0056f9b4be0c1cecca5de896146195ddd6d479addb8384ec3ad39e6d11
Oct 30 04:56:11.149009 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:11.148982    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"]
Oct 30 04:56:11.713100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:11.713050    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2" event={"ID":"0e57e6b3-15af-4562-87de-48093607a0f4","Type":"ContainerStarted","Data":"1763d8a7d3f57ca8ae045ed7c06a1be1817610c344e24e49f86f3742a43ab7d7"}
Oct 30 04:56:11.713604 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:11.713123    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2" event={"ID":"0e57e6b3-15af-4562-87de-48093607a0f4","Type":"ContainerStarted","Data":"8f326d0056f9b4be0c1cecca5de896146195ddd6d479addb8384ec3ad39e6d11"}
Oct 30 04:56:11.757803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:11.757693    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2" podStartSLOduration=1.757677476 podStartE2EDuration="1.757677476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:11.754736188 +0000 UTC m=+3551.496159285" watchObservedRunningTime="2026-10-30 04:56:11.757677476 +0000 UTC m=+3551.499100574"
Oct 30 04:56:11.763308 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:11.763277    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:56:12.723418 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:12.723380    5072 generic.go:356] "Generic (PLEG): container finished" podID="0e57e6b3-15af-4562-87de-48093607a0f4" containerID="1763d8a7d3f57ca8ae045ed7c06a1be1817610c344e24e49f86f3742a43ab7d7" exitCode=0
Oct 30 04:56:12.723845 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:12.723464    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2" event={"ID":"0e57e6b3-15af-4562-87de-48093607a0f4","Type":"ContainerDied","Data":"1763d8a7d3f57ca8ae045ed7c06a1be1817610c344e24e49f86f3742a43ab7d7"}
Oct 30 04:56:13.307426 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.307392    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"]
Oct 30 04:56:13.312363 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.312341    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.371650 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.371611    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"]
Oct 30 04:56:13.472580 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.472533    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pd6j\" (UniqueName: \"kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-kube-api-access-6pd6j\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.472801 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.472616    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.472801 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.472689    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.472801 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.472743    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.577737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.577632    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pd6j\" (UniqueName: \"kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-kube-api-access-6pd6j\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.577737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.577714    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.578020 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.577801    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.578020 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.577851    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.580335 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.579735    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.580335 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.580053    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.582990 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.582952    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.602441 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.602413    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pd6j\" (UniqueName: \"kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-kube-api-access-6pd6j\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.635342 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.633260    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:13.911033 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.911005    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:13.953699 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:13.949446    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"]
Oct 30 04:56:14.101102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.100525    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e57e6b3-15af-4562-87de-48093607a0f4-kube-api-access-nk4tx\" (UniqueName: \"kubernetes.io/projected/0e57e6b3-15af-4562-87de-48093607a0f4-kube-api-access-nk4tx\") pod \"0e57e6b3-15af-4562-87de-48093607a0f4\" (UID: \"0e57e6b3-15af-4562-87de-48093607a0f4\") "
Oct 30 04:56:14.101102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.100606    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-openshift-ca\") pod \"0e57e6b3-15af-4562-87de-48093607a0f4\" (UID: \"0e57e6b3-15af-4562-87de-48093607a0f4\") "
Oct 30 04:56:14.101102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.100742    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-xks-ca\") pod \"0e57e6b3-15af-4562-87de-48093607a0f4\" (UID: \"0e57e6b3-15af-4562-87de-48093607a0f4\") "
Oct 30 04:56:14.101102 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.100979    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-openshift-ca" pod "0e57e6b3-15af-4562-87de-48093607a0f4" (UID: "0e57e6b3-15af-4562-87de-48093607a0f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:14.101478 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.101387    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:14.112017 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.106216    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e57e6b3-15af-4562-87de-48093607a0f4-kube-api-access-nk4tx" pod "0e57e6b3-15af-4562-87de-48093607a0f4" (UID: "0e57e6b3-15af-4562-87de-48093607a0f4"). InnerVolumeSpecName "kube-api-access-nk4tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:14.112017 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.110580    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-xks-ca" pod "0e57e6b3-15af-4562-87de-48093607a0f4" (UID: "0e57e6b3-15af-4562-87de-48093607a0f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:56:14.170698 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.170568    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"]
Oct 30 04:56:14.171801 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.171776    5072 cpu_manager.go:414] "containerMap: removing container" podUID="0e57e6b3-15af-4562-87de-48093607a0f4" containerName="revoke-keys"
Oct 30 04:56:14.171801 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.171799    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e57e6b3-15af-4562-87de-48093607a0f4" containerName="revoke-keys"
Oct 30 04:56:14.171965 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.171948    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e57e6b3-15af-4562-87de-48093607a0f4" containerName="revoke-keys"
Oct 30 04:56:14.178513 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.178485    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.197579 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.197547    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"]
Oct 30 04:56:14.231234 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.225351    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.231234 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.225468    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78cth\" (UniqueName: \"kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-kube-api-access-78cth\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.231234 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.225566    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.231234 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.225718    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.231234 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.225858    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk4tx\" (UniqueName: \"kubernetes.io/projected/0e57e6b3-15af-4562-87de-48093607a0f4-kube-api-access-nk4tx\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:14.231234 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.225879    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e57e6b3-15af-4562-87de-48093607a0f4-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:14.327760 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.327718    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.327960 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.327845    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.327960 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.327913    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.328138 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.327983    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78cth\" (UniqueName: \"kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-kube-api-access-78cth\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.328205 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:14.328155    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b195d0/agc-cred-7b3035355963: secret "agc-cred-7b3035355963" not found
Oct 30 04:56:14.328205 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:14.328175    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z: secret "agc-cred-7b3035355963" not found
Oct 30 04:56:14.328323 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:14.328243    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials podName:51311b6c-7dc0-4dd1-b654-90ade35418db nodeName:}" failed. No retries permitted until 2026-10-30 04:56:14.828227232 +0000 UTC m=+3554.569650307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z" (UID: "51311b6c-7dc0-4dd1-b654-90ade35418db") : secret "agc-cred-7b3035355963" not found
Oct 30 04:56:14.328663 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.328623    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.328783 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.328762    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.381810 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.380124    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78cth\" (UniqueName: \"kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-kube-api-access-78cth\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.644481 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.644122    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"]
Oct 30 04:56:14.648760 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.648726    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"]
Oct 30 04:56:14.743651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.743566    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"
Oct 30 04:56:14.743651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.743595    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2" event={"ID":"0e57e6b3-15af-4562-87de-48093607a0f4","Type":"ContainerDied","Data":"8f326d0056f9b4be0c1cecca5de896146195ddd6d479addb8384ec3ad39e6d11"}
Oct 30 04:56:14.743651 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.743646    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f326d0056f9b4be0c1cecca5de896146195ddd6d479addb8384ec3ad39e6d11"
Oct 30 04:56:14.746560 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.746525    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" event={"ID":"b951c975-edaa-470d-a72f-602fa5d965f4","Type":"ContainerStarted","Data":"943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72"}
Oct 30 04:56:14.746703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.746590    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" event={"ID":"b951c975-edaa-470d-a72f-602fa5d965f4","Type":"ContainerStarted","Data":"932d25ef7129531478202bf012bc8bc2c50d82eb7f2d7b9ecd4d7553e2bb59fb"}
Oct 30 04:56:14.746703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.746623    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:14.746825 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.746706    5072 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" secret="" err="secret \"payload-processing-external-model-e2e--545d7aee-dockercfg-cpcvf\" not found"
Oct 30 04:56:14.790888 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.790832    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" podStartSLOduration=1.790818247 podStartE2EDuration="1.790818247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:14.787816632 +0000 UTC m=+3554.529239732" watchObservedRunningTime="2026-10-30 04:56:14.790818247 +0000 UTC m=+3554.532241344"
Oct 30 04:56:14.834949 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:14.834911    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:14.835153 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:14.835120    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b195d0/agc-cred-7b3035355963: secret "agc-cred-7b3035355963" not found
Oct 30 04:56:14.835153 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:14.835140    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z: secret "agc-cred-7b3035355963" not found
Oct 30 04:56:14.835302 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:14.835200    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials podName:51311b6c-7dc0-4dd1-b654-90ade35418db nodeName:}" failed. No retries permitted until 2026-10-30 04:56:15.835179534 +0000 UTC m=+3555.576602628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z" (UID: "51311b6c-7dc0-4dd1-b654-90ade35418db") : secret "agc-cred-7b3035355963" not found
Oct 30 04:56:14.835302 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:14.835245    5072 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-plugins-e2e-extmod-a-05bee8: configmap "payload-processing-external-model-plugins-e2e-extmod-a-05bee8" not found
Oct 30 04:56:14.835444 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:14.835303    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume podName:b951c975-edaa-470d-a72f-602fa5d965f4 nodeName:}" failed. No retries permitted until 2026-10-30 04:56:15.335286305 +0000 UTC m=+3555.076709394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" (UID: "b951c975-edaa-470d-a72f-602fa5d965f4") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-05bee8" not found
Oct 30 04:56:14.835444 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:14.835394    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-05bee8/agc-cred-569a092ced16: secret "agc-cred-569a092ced16" not found
Oct 30 04:56:14.835444 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:14.835408    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x: secret "agc-cred-569a092ced16" not found
Oct 30 04:56:14.835573 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:14.835455    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials podName:b951c975-edaa-470d-a72f-602fa5d965f4 nodeName:}" failed. No retries permitted until 2026-10-30 04:56:15.335442853 +0000 UTC m=+3555.076865931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" (UID: "b951c975-edaa-470d-a72f-602fa5d965f4") : secret "agc-cred-569a092ced16" not found
Oct 30 04:56:15.351427 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:15.351389    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-05bee8/agc-cred-569a092ced16: secret "agc-cred-569a092ced16" not found
Oct 30 04:56:15.351427 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:15.351419    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x: secret "agc-cred-569a092ced16" not found
Oct 30 04:56:15.351881 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:15.351429    5072 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-plugins-e2e-extmod-a-05bee8: configmap "payload-processing-external-model-plugins-e2e-extmod-a-05bee8" not found
Oct 30 04:56:15.351881 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:15.351474    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials podName:b951c975-edaa-470d-a72f-602fa5d965f4 nodeName:}" failed. No retries permitted until 2026-10-30 04:56:16.3514598 +0000 UTC m=+3556.092882890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" (UID: "b951c975-edaa-470d-a72f-602fa5d965f4") : secret "agc-cred-569a092ced16" not found
Oct 30 04:56:15.351881 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:15.351499    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume podName:b951c975-edaa-470d-a72f-602fa5d965f4 nodeName:}" failed. No retries permitted until 2026-10-30 04:56:16.351482444 +0000 UTC m=+3556.092905541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" (UID: "b951c975-edaa-470d-a72f-602fa5d965f4") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-05bee8" not found
Oct 30 04:56:15.860471 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:15.860422    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:15.860668 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:15.860582    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b195d0/agc-cred-7b3035355963: secret "agc-cred-7b3035355963" not found
Oct 30 04:56:15.860668 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:15.860604    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z: secret "agc-cred-7b3035355963" not found
Oct 30 04:56:15.860668 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:15.860662    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials podName:51311b6c-7dc0-4dd1-b654-90ade35418db nodeName:}" failed. No retries permitted until 2026-10-30 04:56:17.860644684 +0000 UTC m=+3557.602067759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z" (UID: "51311b6c-7dc0-4dd1-b654-90ade35418db") : secret "agc-cred-7b3035355963" not found
Oct 30 04:56:16.369056 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:16.369022    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-05bee8/agc-cred-569a092ced16: secret "agc-cred-569a092ced16" not found
Oct 30 04:56:16.369056 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:16.369053    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x: secret "agc-cred-569a092ced16" not found
Oct 30 04:56:16.369622 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:16.369058    5072 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-plugins-e2e-extmod-a-05bee8: configmap "payload-processing-external-model-plugins-e2e-extmod-a-05bee8" not found
Oct 30 04:56:16.369622 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:16.369149    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials podName:b951c975-edaa-470d-a72f-602fa5d965f4 nodeName:}" failed. No retries permitted until 2026-10-30 04:56:18.369126775 +0000 UTC m=+3558.110549849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" (UID: "b951c975-edaa-470d-a72f-602fa5d965f4") : secret "agc-cred-569a092ced16" not found
Oct 30 04:56:16.369622 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:16.369165    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume podName:b951c975-edaa-470d-a72f-602fa5d965f4 nodeName:}" failed. No retries permitted until 2026-10-30 04:56:18.369158339 +0000 UTC m=+3558.110581414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" (UID: "b951c975-edaa-470d-a72f-602fa5d965f4") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-05bee8" not found
Oct 30 04:56:16.444690 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.444639    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"]
Oct 30 04:56:16.445571 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:16.445131    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z" podUID="51311b6c-7dc0-4dd1-b654-90ade35418db"
Oct 30 04:56:16.461501 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.448267    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"]
Oct 30 04:56:16.629573 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.629406    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"]
Oct 30 04:56:16.635485 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.635458    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.654668 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.652386    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5f1508\"/\"default-dockercfg-t5qrm\""
Oct 30 04:56:16.654668 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.652539    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5f1508\"/\"openshift-service-ca.crt\""
Oct 30 04:56:16.654668 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.652800    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5f1508\"/\"test-model-5f1508-kserve-self-signed-certs\""
Oct 30 04:56:16.654668 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.652857    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5f1508\"/\"kube-root-ca.crt\""
Oct 30 04:56:16.655646 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.655624    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"]
Oct 30 04:56:16.673746 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.673705    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac55f48a-97c9-4178-8e54-8b12063986e6-tls-certs\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.673928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.673791    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-home\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.673928 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.673832    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj2cg\" (UniqueName: \"kubernetes.io/projected/ac55f48a-97c9-4178-8e54-8b12063986e6-kube-api-access-zj2cg\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.674019 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.673952    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-tmp-dir\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.674105 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.674069    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-model-cache\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.674225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.674209    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-dshm\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.764572 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.764544    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:16.776229 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.776194    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac55f48a-97c9-4178-8e54-8b12063986e6-tls-certs\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.776391 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.776270    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-home\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.776391 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.776310    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2cg\" (UniqueName: \"kubernetes.io/projected/ac55f48a-97c9-4178-8e54-8b12063986e6-kube-api-access-zj2cg\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.776535 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.776424    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-tmp-dir\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.776600 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.776545    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-model-cache\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.776679 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.776657    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-dshm\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.777311 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.777289    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-home\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.777480 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.777311    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-model-cache\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.777939 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.777923    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-tmp-dir\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.786745 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.779523    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-dshm\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.787550 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.787520    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac55f48a-97c9-4178-8e54-8b12063986e6-tls-certs\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.804235 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.801804    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2cg\" (UniqueName: \"kubernetes.io/projected/ac55f48a-97c9-4178-8e54-8b12063986e6-kube-api-access-zj2cg\") pod \"test-model-5f1508-kserve-9cd4cfb5-kf9f4\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") " pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.825121 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.825088    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:16.877166 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.877125    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-kube-api-access-78cth\" (UniqueName: \"kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-kube-api-access-78cth\") pod \"51311b6c-7dc0-4dd1-b654-90ade35418db\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") "
Oct 30 04:56:16.877413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.877399    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-plugins-config-volume\") pod \"51311b6c-7dc0-4dd1-b654-90ade35418db\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") "
Oct 30 04:56:16.877484 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.877437    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-routing-overlay-volume\") pod \"51311b6c-7dc0-4dd1-b654-90ade35418db\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") "
Oct 30 04:56:16.878145 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.878051    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-routing-overlay-volume" pod "51311b6c-7dc0-4dd1-b654-90ade35418db" (UID: "51311b6c-7dc0-4dd1-b654-90ade35418db"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:16.878497 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.878456    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-plugins-config-volume" pod "51311b6c-7dc0-4dd1-b654-90ade35418db" (UID: "51311b6c-7dc0-4dd1-b654-90ade35418db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:16.881296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.881239    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-kube-api-access-78cth" pod "51311b6c-7dc0-4dd1-b654-90ade35418db" (UID: "51311b6c-7dc0-4dd1-b654-90ade35418db"). InnerVolumeSpecName "kube-api-access-78cth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:16.967400 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.967356    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:16.981452 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.981410    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:16.981452 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.981453    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51311b6c-7dc0-4dd1-b654-90ade35418db-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:16.981726 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:16.981468    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78cth\" (UniqueName: \"kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-kube-api-access-78cth\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:17.385676 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:56:17.385645    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac55f48a_97c9_4178_8e54_8b12063986e6.slice/crio-d3a3e1b0b20f76d7254be43710e05c44a3e6acab1d0f2d2429e24f89fb3af8a7 WatchSource:0}: Error finding container d3a3e1b0b20f76d7254be43710e05c44a3e6acab1d0f2d2429e24f89fb3af8a7: Status 404 returned error can't find the container with id d3a3e1b0b20f76d7254be43710e05c44a3e6acab1d0f2d2429e24f89fb3af8a7
Oct 30 04:56:17.387537 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:17.387512    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"]
Oct 30 04:56:17.776751 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:17.776712    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4" event={"ID":"ac55f48a-97c9-4178-8e54-8b12063986e6","Type":"ContainerStarted","Data":"2e5362e0ddf8c4a163c5a56b9e3d20c94cc1c9dae2a83e7e1e1ef53f8e0db51d"}
Oct 30 04:56:17.776954 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:17.776768    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4" event={"ID":"ac55f48a-97c9-4178-8e54-8b12063986e6","Type":"ContainerStarted","Data":"d3a3e1b0b20f76d7254be43710e05c44a3e6acab1d0f2d2429e24f89fb3af8a7"}
Oct 30 04:56:17.776954 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:17.776804    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:17.825097 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:17.825031    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4" podStartSLOduration=1.825016905 podStartE2EDuration="1.825016905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:17.82067094 +0000 UTC m=+3557.562094049" watchObservedRunningTime="2026-10-30 04:56:17.825016905 +0000 UTC m=+3557.566440002"
Oct 30 04:56:17.894175 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:17.894138    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"]
Oct 30 04:56:17.894885 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:17.894860    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"]
Oct 30 04:56:17.912550 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:17.912506    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z\" (UID: \"51311b6c-7dc0-4dd1-b654-90ade35418db\") " pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z"
Oct 30 04:56:17.912739 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:17.912717    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b195d0/agc-cred-7b3035355963: secret "agc-cred-7b3035355963" not found
Oct 30 04:56:17.912841 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:17.912742    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z: secret "agc-cred-7b3035355963" not found
Oct 30 04:56:17.912841 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:17.912802    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials podName:51311b6c-7dc0-4dd1-b654-90ade35418db nodeName:}" failed. No retries permitted until 2026-10-30 04:56:21.912783091 +0000 UTC m=+3561.654206181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b195d0-86f4f2q9z" (UID: "51311b6c-7dc0-4dd1-b654-90ade35418db") : secret "agc-cred-7b3035355963" not found
Oct 30 04:56:17.967864 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:17.967833    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:18.013849 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:18.013812    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51311b6c-7dc0-4dd1-b654-90ade35418db-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:18.410590 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:18.410556    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51311b6c-7dc0-4dd1-b654-90ade35418db" path="/var/lib/kubelet/pods/51311b6c-7dc0-4dd1-b654-90ade35418db/volumes"
Oct 30 04:56:18.419547 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:18.419515    5072 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-05bee8/agc-cred-569a092ced16: secret "agc-cred-569a092ced16" not found
Oct 30 04:56:18.419547 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:18.419544    5072 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x: secret "agc-cred-569a092ced16" not found
Oct 30 04:56:18.419775 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:18.419572    5072 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-plugins-e2e-extmod-a-05bee8: configmap "payload-processing-external-model-plugins-e2e-extmod-a-05bee8" not found
Oct 30 04:56:18.419775 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:18.419615    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials podName:b951c975-edaa-470d-a72f-602fa5d965f4 nodeName:}" failed. No retries permitted until 2026-10-30 04:56:22.419595111 +0000 UTC m=+3562.161018208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" (UID: "b951c975-edaa-470d-a72f-602fa5d965f4") : secret "agc-cred-569a092ced16" not found
Oct 30 04:56:18.419775 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:18.419635    5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume podName:b951c975-edaa-470d-a72f-602fa5d965f4 nodeName:}" failed. No retries permitted until 2026-10-30 04:56:22.419623141 +0000 UTC m=+3562.161046216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" (UID: "b951c975-edaa-470d-a72f-602fa5d965f4") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-05bee8" not found
Oct 30 04:56:19.645316 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:19.645267    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9" podUID="75c9a69b-9bcb-4b1c-9697-6ce978ea4779" containerName="payload-processing" containerID="cri-o://2b98fd88bda9c63b658f0e041e2abedd485384f3aeaabc046195bff2054b8745" gracePeriod=25
Oct 30 04:56:19.795328 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:19.795291    5072 generic.go:356] "Generic (PLEG): container finished" podID="75c9a69b-9bcb-4b1c-9697-6ce978ea4779" containerID="2b98fd88bda9c63b658f0e041e2abedd485384f3aeaabc046195bff2054b8745" exitCode=0
Oct 30 04:56:19.795506 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:19.795366    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9" event={"ID":"75c9a69b-9bcb-4b1c-9697-6ce978ea4779","Type":"ContainerDied","Data":"2b98fd88bda9c63b658f0e041e2abedd485384f3aeaabc046195bff2054b8745"}
Oct 30 04:56:19.932516 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:19.932489    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:56:20.042109 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.042035    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-plugins-config-volume\") pod \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") "
Oct 30 04:56:20.042306 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.042145    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-kube-api-access-2xwvh\" (UniqueName: \"kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-kube-api-access-2xwvh\") pod \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") "
Oct 30 04:56:20.042306 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.042270    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-routing-overlay-volume\") pod \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") "
Oct 30 04:56:20.042439 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.042314    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-provider-credentials\" (UniqueName: \"kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-provider-credentials\") pod \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\" (UID: \"75c9a69b-9bcb-4b1c-9697-6ce978ea4779\") "
Oct 30 04:56:20.042520 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.042488    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-plugins-config-volume" pod "75c9a69b-9bcb-4b1c-9697-6ce978ea4779" (UID: "75c9a69b-9bcb-4b1c-9697-6ce978ea4779"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:20.042676 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.042650    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-routing-overlay-volume" pod "75c9a69b-9bcb-4b1c-9697-6ce978ea4779" (UID: "75c9a69b-9bcb-4b1c-9697-6ce978ea4779"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:20.043210 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.043188    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:20.043306 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.043215    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:20.044619 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.044599    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-kube-api-access-2xwvh" pod "75c9a69b-9bcb-4b1c-9697-6ce978ea4779" (UID: "75c9a69b-9bcb-4b1c-9697-6ce978ea4779"). InnerVolumeSpecName "kube-api-access-2xwvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:20.044757 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.044744    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-provider-credentials" pod "75c9a69b-9bcb-4b1c-9697-6ce978ea4779" (UID: "75c9a69b-9bcb-4b1c-9697-6ce978ea4779"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:20.144250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.144220    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xwvh\" (UniqueName: \"kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-kube-api-access-2xwvh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:20.144250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.144246    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75c9a69b-9bcb-4b1c-9697-6ce978ea4779-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:20.754243 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.754185    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" podUID="b951c975-edaa-470d-a72f-602fa5d965f4" containerName="payload-processing" containerID="cri-o://943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72" gracePeriod=25
Oct 30 04:56:20.806918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.806886    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"
Oct 30 04:56:20.806918 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.806894    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9" event={"ID":"75c9a69b-9bcb-4b1c-9697-6ce978ea4779","Type":"ContainerDied","Data":"340f3e2508bd4a57d526e089cefd23180d53e9aed59e081a95790486d338f0ef"}
Oct 30 04:56:20.807156 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.806964    5072 scope.go:122] "RemoveContainer" containerID="2b98fd88bda9c63b658f0e041e2abedd485384f3aeaabc046195bff2054b8745"
Oct 30 04:56:20.872840 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.872811    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"]
Oct 30 04:56:20.885429 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:20.885401    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-59f7s2ng9"]
Oct 30 04:56:21.039506 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.039480    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:21.160238 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.160199    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume\") pod \"b951c975-edaa-470d-a72f-602fa5d965f4\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") "
Oct 30 04:56:21.160443 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.160396    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials\") pod \"b951c975-edaa-470d-a72f-602fa5d965f4\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") "
Oct 30 04:56:21.160542 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.160525    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-kube-api-access-6pd6j\" (UniqueName: \"kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-kube-api-access-6pd6j\") pod \"b951c975-edaa-470d-a72f-602fa5d965f4\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") "
Oct 30 04:56:21.160630 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.160617    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-routing-overlay-volume\") pod \"b951c975-edaa-470d-a72f-602fa5d965f4\" (UID: \"b951c975-edaa-470d-a72f-602fa5d965f4\") "
Oct 30 04:56:21.160701 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.160644    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume" pod "b951c975-edaa-470d-a72f-602fa5d965f4" (UID: "b951c975-edaa-470d-a72f-602fa5d965f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:21.161016 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.160991    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-routing-overlay-volume" pod "b951c975-edaa-470d-a72f-602fa5d965f4" (UID: "b951c975-edaa-470d-a72f-602fa5d965f4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:21.161186 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.161160    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:21.161186 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.161185    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b951c975-edaa-470d-a72f-602fa5d965f4-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:21.162689 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.162664    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials" pod "b951c975-edaa-470d-a72f-602fa5d965f4" (UID: "b951c975-edaa-470d-a72f-602fa5d965f4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:21.162780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.162764    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-kube-api-access-6pd6j" pod "b951c975-edaa-470d-a72f-602fa5d965f4" (UID: "b951c975-edaa-470d-a72f-602fa5d965f4"). InnerVolumeSpecName "kube-api-access-6pd6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:21.197846 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.197818    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"]
Oct 30 04:56:21.198732 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.198717    5072 cpu_manager.go:414] "containerMap: removing container" podUID="b951c975-edaa-470d-a72f-602fa5d965f4" containerName="payload-processing"
Oct 30 04:56:21.198795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.198734    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b951c975-edaa-470d-a72f-602fa5d965f4" containerName="payload-processing"
Oct 30 04:56:21.198795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.198751    5072 cpu_manager.go:414] "containerMap: removing container" podUID="75c9a69b-9bcb-4b1c-9697-6ce978ea4779" containerName="payload-processing"
Oct 30 04:56:21.198795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.198756    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c9a69b-9bcb-4b1c-9697-6ce978ea4779" containerName="payload-processing"
Oct 30 04:56:21.198903 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.198893    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c9a69b-9bcb-4b1c-9697-6ce978ea4779" containerName="payload-processing"
Oct 30 04:56:21.198941 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.198915    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b951c975-edaa-470d-a72f-602fa5d965f4" containerName="payload-processing"
Oct 30 04:56:21.205185 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.205166    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:21.220984 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.220411    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:56:21.221351 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.221318    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:56:21.221642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.221622    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"]
Oct 30 04:56:21.262379 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.262347    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pd6j\" (UniqueName: \"kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-kube-api-access-6pd6j\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:21.262379 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.262375    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b951c975-edaa-470d-a72f-602fa5d965f4-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:21.363774 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.363670    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af82feed-wprqp\" (UID: \"2775b776-88da-49f2-81ad-2bd8a68b3023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:21.363934 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.363857    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af82feed-wprqp\" (UID: \"2775b776-88da-49f2-81ad-2bd8a68b3023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:21.363934 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.363919    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6dlw\" (UniqueName: \"kubernetes.io/projected/2775b776-88da-49f2-81ad-2bd8a68b3023-kube-api-access-g6dlw\") pod \"maas-api-revoke-sub-af82feed-wprqp\" (UID: \"2775b776-88da-49f2-81ad-2bd8a68b3023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:21.449163 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.449117    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw" podUID="4688132f-c53c-4441-83a7-7d0947d554bc" containerName="payload-processing" containerID="cri-o://7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151" gracePeriod=25
Oct 30 04:56:21.465400 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.465366    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af82feed-wprqp\" (UID: \"2775b776-88da-49f2-81ad-2bd8a68b3023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:21.465522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.465498    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af82feed-wprqp\" (UID: \"2775b776-88da-49f2-81ad-2bd8a68b3023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:21.465587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.465539    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6dlw\" (UniqueName: \"kubernetes.io/projected/2775b776-88da-49f2-81ad-2bd8a68b3023-kube-api-access-g6dlw\") pod \"maas-api-revoke-sub-af82feed-wprqp\" (UID: \"2775b776-88da-49f2-81ad-2bd8a68b3023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:21.466165 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.466141    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af82feed-wprqp\" (UID: \"2775b776-88da-49f2-81ad-2bd8a68b3023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:21.467964 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.467940    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af82feed-wprqp\" (UID: \"2775b776-88da-49f2-81ad-2bd8a68b3023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:21.489903 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.489872    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6dlw\" (UniqueName: \"kubernetes.io/projected/2775b776-88da-49f2-81ad-2bd8a68b3023-kube-api-access-g6dlw\") pod \"maas-api-revoke-sub-af82feed-wprqp\" (UID: \"2775b776-88da-49f2-81ad-2bd8a68b3023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:21.524538 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.524508    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:21.741402 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.741376    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:56:21.817489 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.817459    5072 generic.go:356] "Generic (PLEG): container finished" podID="b951c975-edaa-470d-a72f-602fa5d965f4" containerID="943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72" exitCode=0
Oct 30 04:56:21.817954 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.817516    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"
Oct 30 04:56:21.817954 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.817539    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" event={"ID":"b951c975-edaa-470d-a72f-602fa5d965f4","Type":"ContainerDied","Data":"943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72"}
Oct 30 04:56:21.817954 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.817603    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x" event={"ID":"b951c975-edaa-470d-a72f-602fa5d965f4","Type":"ContainerDied","Data":"932d25ef7129531478202bf012bc8bc2c50d82eb7f2d7b9ecd4d7553e2bb59fb"}
Oct 30 04:56:21.817954 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.817632    5072 scope.go:122] "RemoveContainer" containerID="943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72"
Oct 30 04:56:21.845898 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.845859    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"]
Oct 30 04:56:21.848856 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:56:21.848830    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2775b776_88da_49f2_81ad_2bd8a68b3023.slice/crio-47695b916a45c534da023bf51d007528f2b9279368b4a07fafffcefeb01a5ac3 WatchSource:0}: Error finding container 47695b916a45c534da023bf51d007528f2b9279368b4a07fafffcefeb01a5ac3: Status 404 returned error can't find the container with id 47695b916a45c534da023bf51d007528f2b9279368b4a07fafffcefeb01a5ac3
Oct 30 04:56:21.849921 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.849894    5072 generic.go:356] "Generic (PLEG): container finished" podID="4688132f-c53c-4441-83a7-7d0947d554bc" containerID="7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151" exitCode=0
Oct 30 04:56:21.850043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.849939    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"
Oct 30 04:56:21.850043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.849968    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw" event={"ID":"4688132f-c53c-4441-83a7-7d0947d554bc","Type":"ContainerDied","Data":"7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151"}
Oct 30 04:56:21.850043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.850019    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw" event={"ID":"4688132f-c53c-4441-83a7-7d0947d554bc","Type":"ContainerDied","Data":"ab9822746b7df32788c7858f3b295b3759e46c37cb8e690943b8d31d8dde17f8"}
Oct 30 04:56:21.857431 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.857406    5072 scope.go:122] "RemoveContainer" containerID="943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72"
Oct 30 04:56:21.857769 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:21.857741    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72\": container with ID starting with 943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72 not found: ID does not exist" containerID="943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72"
Oct 30 04:56:21.857884 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.857777    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72"} err="failed to get container status \"943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72\": rpc error: code = NotFound desc = could not find container \"943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72\": container with ID starting with 943f2629ada2deef7103173da315720d1cad2b72fec774a2a5b62b5ac3a8df72 not found: ID does not exist"
Oct 30 04:56:21.857884 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.857803    5072 scope.go:122] "RemoveContainer" containerID="7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151"
Oct 30 04:56:21.869835 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.869801    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-routing-overlay-volume\") pod \"4688132f-c53c-4441-83a7-7d0947d554bc\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") "
Oct 30 04:56:21.869932 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.869859    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-kube-api-access-z8chk\" (UniqueName: \"kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-kube-api-access-z8chk\") pod \"4688132f-c53c-4441-83a7-7d0947d554bc\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") "
Oct 30 04:56:21.870056 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.870042    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-provider-credentials\") pod \"4688132f-c53c-4441-83a7-7d0947d554bc\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") "
Oct 30 04:56:21.870162 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.870107    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-plugins-config-volume\") pod \"4688132f-c53c-4441-83a7-7d0947d554bc\" (UID: \"4688132f-c53c-4441-83a7-7d0947d554bc\") "
Oct 30 04:56:21.870222 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.870181    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-routing-overlay-volume" pod "4688132f-c53c-4441-83a7-7d0947d554bc" (UID: "4688132f-c53c-4441-83a7-7d0947d554bc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:21.870491 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.870469    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-plugins-config-volume" pod "4688132f-c53c-4441-83a7-7d0947d554bc" (UID: "4688132f-c53c-4441-83a7-7d0947d554bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:21.870587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.870557    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:21.870587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.870570    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4688132f-c53c-4441-83a7-7d0947d554bc-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:21.872508 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.872479    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-kube-api-access-z8chk" pod "4688132f-c53c-4441-83a7-7d0947d554bc" (UID: "4688132f-c53c-4441-83a7-7d0947d554bc"). InnerVolumeSpecName "kube-api-access-z8chk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:21.873430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.873397    5072 scope.go:122] "RemoveContainer" containerID="7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151"
Oct 30 04:56:21.873624 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.873602    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-provider-credentials" pod "4688132f-c53c-4441-83a7-7d0947d554bc" (UID: "4688132f-c53c-4441-83a7-7d0947d554bc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:21.873755 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:21.873729    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151\": container with ID starting with 7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151 not found: ID does not exist" containerID="7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151"
Oct 30 04:56:21.873806 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.873765    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151"} err="failed to get container status \"7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151\": rpc error: code = NotFound desc = could not find container \"7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151\": container with ID starting with 7587ab0eab3d460fbf9f3d1f2a5d909f333efb0b96706a3d138f92b2e1922151 not found: ID does not exist"
Oct 30 04:56:21.907375 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.907320    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"]
Oct 30 04:56:21.913058 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.913036    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-7d654bz9x"]
Oct 30 04:56:21.972430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.972396    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:21.972430 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:21.972432    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8chk\" (UniqueName: \"kubernetes.io/projected/4688132f-c53c-4441-83a7-7d0947d554bc-kube-api-access-z8chk\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:22.207189 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.207153    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"]
Oct 30 04:56:22.214069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.214040    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b195d0/payload-processing-external-model-e2e-extmod-b-b195d0-6d8d7tjfw"]
Oct 30 04:56:22.299289 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.299255    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"]
Oct 30 04:56:22.300533 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.300513    5072 cpu_manager.go:414] "containerMap: removing container" podUID="4688132f-c53c-4441-83a7-7d0947d554bc" containerName="payload-processing"
Oct 30 04:56:22.300611 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.300537    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4688132f-c53c-4441-83a7-7d0947d554bc" containerName="payload-processing"
Oct 30 04:56:22.300815 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.300803    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="4688132f-c53c-4441-83a7-7d0947d554bc" containerName="payload-processing"
Oct 30 04:56:22.305810 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.305786    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.328654 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.328625    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7700a7-openshift-default-dockercfg-rrfpz\""
Oct 30 04:56:22.334995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.334968    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"]
Oct 30 04:56:22.377872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.377532    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-certs\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.377872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.377581    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a006882-7f2d-4624-9bab-4d3623a0b75e-istiod-ca-cert\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.377872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.377655    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-data\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.377872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.377714    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-podinfo\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.377872 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.377756    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-credential-socket\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.378316 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.377884    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-socket\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.378316 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.377908    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-token\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.378316 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.377937    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-envoy\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.378316 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.377983    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztr7p\" (UniqueName: \"kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-kube-api-access-ztr7p\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.415142 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.415034    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4688132f-c53c-4441-83a7-7d0947d554bc" path="/var/lib/kubelet/pods/4688132f-c53c-4441-83a7-7d0947d554bc/volumes"
Oct 30 04:56:22.415529 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.415510    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c9a69b-9bcb-4b1c-9697-6ce978ea4779" path="/var/lib/kubelet/pods/75c9a69b-9bcb-4b1c-9697-6ce978ea4779/volumes"
Oct 30 04:56:22.415940 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.415923    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b951c975-edaa-470d-a72f-602fa5d965f4" path="/var/lib/kubelet/pods/b951c975-edaa-470d-a72f-602fa5d965f4/volumes"
Oct 30 04:56:22.489139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.486121    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-podinfo\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.489139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.486203    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-credential-socket\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.489139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.486348    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-socket\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.489139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.486373    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-token\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.489139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.486404    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-envoy\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.489139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.486452    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr7p\" (UniqueName: \"kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-kube-api-access-ztr7p\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.489139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.486548    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-certs\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.489139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.486581    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a006882-7f2d-4624-9bab-4d3623a0b75e-istiod-ca-cert\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.489139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.486667    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-data\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.489139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.487193    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-data\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.489139 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.488196    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-certs\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.490464 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.490432    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a006882-7f2d-4624-9bab-4d3623a0b75e-istiod-ca-cert\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.490602 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.490497    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-socket\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.490768 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.490682    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-credential-socket\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.492656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.492626    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-envoy\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.493243 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.493217    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-podinfo\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.548100 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.523794    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztr7p\" (UniqueName: \"kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-kube-api-access-ztr7p\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.558810 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.558762    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-token\") pod \"e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") " pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.634101 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.633423    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:22.863050 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.863014    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp" event={"ID":"2775b776-88da-49f2-81ad-2bd8a68b3023","Type":"ContainerStarted","Data":"b753111e05e32220613dca70e6d89e6b39555567d35b7504fe9062c261959bad"}
Oct 30 04:56:22.863579 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.863069    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp" event={"ID":"2775b776-88da-49f2-81ad-2bd8a68b3023","Type":"ContainerStarted","Data":"47695b916a45c534da023bf51d007528f2b9279368b4a07fafffcefeb01a5ac3"}
Oct 30 04:56:22.905622 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.905563    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp" podStartSLOduration=1.905547724 podStartE2EDuration="1.905547724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:22.903965201 +0000 UTC m=+3562.645388300" watchObservedRunningTime="2026-10-30 04:56:22.905547724 +0000 UTC m=+3562.646970822"
Oct 30 04:56:22.990127 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:22.990096    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"]
Oct 30 04:56:23.041046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:23.039641    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:56:23.041046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:23.039784    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:56:23.041046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:23.039867    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:56:23.876528 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:23.876493    5072 generic.go:356] "Generic (PLEG): container finished" podID="2775b776-88da-49f2-81ad-2bd8a68b3023" containerID="b753111e05e32220613dca70e6d89e6b39555567d35b7504fe9062c261959bad" exitCode=0
Oct 30 04:56:23.876977 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:23.876570    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp" event={"ID":"2775b776-88da-49f2-81ad-2bd8a68b3023","Type":"ContainerDied","Data":"b753111e05e32220613dca70e6d89e6b39555567d35b7504fe9062c261959bad"}
Oct 30 04:56:23.878455 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:23.878415    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" event={"ID":"1a006882-7f2d-4624-9bab-4d3623a0b75e","Type":"ContainerStarted","Data":"764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501"}
Oct 30 04:56:23.878589 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:23.878483    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" event={"ID":"1a006882-7f2d-4624-9bab-4d3623a0b75e","Type":"ContainerStarted","Data":"982e803d449f1cd12e5380c4ff3286d126f87d8a1b20140b2376d8a56c82de4e"}
Oct 30 04:56:23.970703 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:23.970645    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" podStartSLOduration=1.9706300780000001 podStartE2EDuration="1.970630078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:23.968417083 +0000 UTC m=+3563.709840194" watchObservedRunningTime="2026-10-30 04:56:23.970630078 +0000 UTC m=+3563.712053245"
Oct 30 04:56:24.634835 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:24.634800    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:24.641414 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:24.641390    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:24.887669 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:24.887579    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:24.888635 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:24.888619    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:56:25.111642 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.111611    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:25.231371 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.231343    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2775b776-88da-49f2-81ad-2bd8a68b3023-kube-api-access-g6dlw\" (UniqueName: \"kubernetes.io/projected/2775b776-88da-49f2-81ad-2bd8a68b3023-kube-api-access-g6dlw\") pod \"2775b776-88da-49f2-81ad-2bd8a68b3023\" (UID: \"2775b776-88da-49f2-81ad-2bd8a68b3023\") "
Oct 30 04:56:25.231555 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.231390    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-openshift-ca\") pod \"2775b776-88da-49f2-81ad-2bd8a68b3023\" (UID: \"2775b776-88da-49f2-81ad-2bd8a68b3023\") "
Oct 30 04:56:25.231555 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.231509    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-xks-ca\") pod \"2775b776-88da-49f2-81ad-2bd8a68b3023\" (UID: \"2775b776-88da-49f2-81ad-2bd8a68b3023\") "
Oct 30 04:56:25.231821 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.231792    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-openshift-ca" pod "2775b776-88da-49f2-81ad-2bd8a68b3023" (UID: "2775b776-88da-49f2-81ad-2bd8a68b3023"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:25.232148 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.232124    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:25.233765 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.233744    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2775b776-88da-49f2-81ad-2bd8a68b3023-kube-api-access-g6dlw" pod "2775b776-88da-49f2-81ad-2bd8a68b3023" (UID: "2775b776-88da-49f2-81ad-2bd8a68b3023"). InnerVolumeSpecName "kube-api-access-g6dlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:25.233896 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.233819    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-xks-ca" pod "2775b776-88da-49f2-81ad-2bd8a68b3023" (UID: "2775b776-88da-49f2-81ad-2bd8a68b3023"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:56:25.334364 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.334336    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6dlw\" (UniqueName: \"kubernetes.io/projected/2775b776-88da-49f2-81ad-2bd8a68b3023-kube-api-access-g6dlw\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:25.334364 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.334364    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2775b776-88da-49f2-81ad-2bd8a68b3023-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:25.900011 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.899983    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"
Oct 30 04:56:25.900522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.900015    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp" event={"ID":"2775b776-88da-49f2-81ad-2bd8a68b3023","Type":"ContainerDied","Data":"47695b916a45c534da023bf51d007528f2b9279368b4a07fafffcefeb01a5ac3"}
Oct 30 04:56:25.900522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:25.900070    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47695b916a45c534da023bf51d007528f2b9279368b4a07fafffcefeb01a5ac3"
Oct 30 04:56:26.968250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:26.968221    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:26.981778 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:26.981744    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:27.930869 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:27.930838    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 04:56:28.296780 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.296750    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"]
Oct 30 04:56:28.297652 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.297637    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2775b776-88da-49f2-81ad-2bd8a68b3023" containerName="revoke-keys"
Oct 30 04:56:28.297714 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.297654    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2775b776-88da-49f2-81ad-2bd8a68b3023" containerName="revoke-keys"
Oct 30 04:56:28.297824 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.297814    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2775b776-88da-49f2-81ad-2bd8a68b3023" containerName="revoke-keys"
Oct 30 04:56:28.302141 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.302116    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"
Oct 30 04:56:28.319226 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.319202    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7700a7\""
Oct 30 04:56:28.319361 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.319244    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7700a7-dockercfg-hpjjc\""
Oct 30 04:56:28.322000 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.321979    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"]
Oct 30 04:56:28.383818 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.383771    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa34040e-b825-4490-bad5-637800da41e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf\" (UID: \"aa34040e-b825-4490-bad5-637800da41e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"
Oct 30 04:56:28.384031 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.383834    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2px6w\" (UniqueName: \"kubernetes.io/projected/aa34040e-b825-4490-bad5-637800da41e2-kube-api-access-2px6w\") pod \"payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf\" (UID: \"aa34040e-b825-4490-bad5-637800da41e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"
Oct 30 04:56:28.494289 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.494233    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa34040e-b825-4490-bad5-637800da41e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf\" (UID: \"aa34040e-b825-4490-bad5-637800da41e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"
Oct 30 04:56:28.494606 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.494315    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2px6w\" (UniqueName: \"kubernetes.io/projected/aa34040e-b825-4490-bad5-637800da41e2-kube-api-access-2px6w\") pod \"payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf\" (UID: \"aa34040e-b825-4490-bad5-637800da41e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"
Oct 30 04:56:28.496809 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.496774    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa34040e-b825-4490-bad5-637800da41e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf\" (UID: \"aa34040e-b825-4490-bad5-637800da41e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"
Oct 30 04:56:28.524352 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.524276    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2px6w\" (UniqueName: \"kubernetes.io/projected/aa34040e-b825-4490-bad5-637800da41e2-kube-api-access-2px6w\") pod \"payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf\" (UID: \"aa34040e-b825-4490-bad5-637800da41e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"
Oct 30 04:56:28.533624 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.533596    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"]
Oct 30 04:56:28.539150 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.539118    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"]
Oct 30 04:56:28.539301 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.539216    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"
Oct 30 04:56:28.596142 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.596105    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45\" (UID: \"7b81fac8-afdb-4c6b-a95a-4654a89d9b31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"
Oct 30 04:56:28.596309 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.596292    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xx9v\" (UniqueName: \"kubernetes.io/projected/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-kube-api-access-6xx9v\") pod \"payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45\" (UID: \"7b81fac8-afdb-4c6b-a95a-4654a89d9b31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"
Oct 30 04:56:28.617933 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.617904    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"
Oct 30 04:56:28.697851 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.697517    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45\" (UID: \"7b81fac8-afdb-4c6b-a95a-4654a89d9b31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"
Oct 30 04:56:28.697851 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.697688    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xx9v\" (UniqueName: \"kubernetes.io/projected/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-kube-api-access-6xx9v\") pod \"payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45\" (UID: \"7b81fac8-afdb-4c6b-a95a-4654a89d9b31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"
Oct 30 04:56:28.698536 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.698509    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45\" (UID: \"7b81fac8-afdb-4c6b-a95a-4654a89d9b31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"
Oct 30 04:56:28.736408 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.736379    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xx9v\" (UniqueName: \"kubernetes.io/projected/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-kube-api-access-6xx9v\") pod \"payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45\" (UID: \"7b81fac8-afdb-4c6b-a95a-4654a89d9b31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"
Oct 30 04:56:28.856055 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.855964    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"
Oct 30 04:56:28.930312 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:56:28.930277    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa34040e_b825_4490_bad5_637800da41e2.slice/crio-9f70b77b644b802e2942a87162a7799bdd8d242d5571ecd1cdb68cfe008b6ce1 WatchSource:0}: Error finding container 9f70b77b644b802e2942a87162a7799bdd8d242d5571ecd1cdb68cfe008b6ce1: Status 404 returned error can't find the container with id 9f70b77b644b802e2942a87162a7799bdd8d242d5571ecd1cdb68cfe008b6ce1
Oct 30 04:56:28.936092 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:28.934054    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"]
Oct 30 04:56:29.164842 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:56:29.164808    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b81fac8_afdb_4c6b_a95a_4654a89d9b31.slice/crio-e7333167e3b31186dc3d6beb3f7e4f6e0048abf2278597813c8a99f2cf31f73a WatchSource:0}: Error finding container e7333167e3b31186dc3d6beb3f7e4f6e0048abf2278597813c8a99f2cf31f73a: Status 404 returned error can't find the container with id e7333167e3b31186dc3d6beb3f7e4f6e0048abf2278597813c8a99f2cf31f73a
Oct 30 04:56:29.165781 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:29.165758    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"]
Oct 30 04:56:29.942855 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:29.942815    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45" event={"ID":"7b81fac8-afdb-4c6b-a95a-4654a89d9b31","Type":"ContainerStarted","Data":"2399637d24a925a3a9b640708d700cca0f44a7923f6bfbec0f9bb21fbdc42334"}
Oct 30 04:56:29.943333 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:29.942885    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"
Oct 30 04:56:29.943333 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:29.942909    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45" event={"ID":"7b81fac8-afdb-4c6b-a95a-4654a89d9b31","Type":"ContainerStarted","Data":"e7333167e3b31186dc3d6beb3f7e4f6e0048abf2278597813c8a99f2cf31f73a"}
Oct 30 04:56:29.944838 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:29.944807    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf" event={"ID":"aa34040e-b825-4490-bad5-637800da41e2","Type":"ContainerStarted","Data":"f735aa6bf10267b66dc3fc63b04555b0935e28f53936a4c6eccd0b9ff5eafe29"}
Oct 30 04:56:29.944990 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:29.944863    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf" event={"ID":"aa34040e-b825-4490-bad5-637800da41e2","Type":"ContainerStarted","Data":"9f70b77b644b802e2942a87162a7799bdd8d242d5571ecd1cdb68cfe008b6ce1"}
Oct 30 04:56:29.944990 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:29.944893    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"
Oct 30 04:56:29.987717 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:29.987660    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45" podStartSLOduration=1.98764735 podStartE2EDuration="1.98764735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:29.984792034 +0000 UTC m=+3569.726215131" watchObservedRunningTime="2026-10-30 04:56:29.98764735 +0000 UTC m=+3569.729070486"
Oct 30 04:56:30.035658 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:30.035608    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf" podStartSLOduration=2.035593995 podStartE2EDuration="2.035593995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:30.032142962 +0000 UTC m=+3569.773566059" watchObservedRunningTime="2026-10-30 04:56:30.035593995 +0000 UTC m=+3569.777017091"
Oct 30 04:56:33.214231 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.214176    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"]
Oct 30 04:56:33.219155 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.219137    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.241140 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.241111    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"]
Oct 30 04:56:33.267399 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.267370    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-84482b258\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.267551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.267406    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cff9n\" (UniqueName: \"kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-kube-api-access-cff9n\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-84482b258\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.267551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.267450    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-84482b258\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.267702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.267554    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-84482b258\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.369170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.369139    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-84482b258\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.369325 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.369253    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-84482b258\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.369446 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.369429    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-84482b258\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.369521 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.369461    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cff9n\" (UniqueName: \"kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-kube-api-access-cff9n\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-84482b258\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.370232 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.370202    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-84482b258\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.370345 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.370317    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-84482b258\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.371899 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.371879    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-84482b258\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.397368 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.396026    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cff9n\" (UniqueName: \"kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-kube-api-access-cff9n\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-84482b258\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.512819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.512728    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"]
Oct 30 04:56:33.517791 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.517769    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:33.533272 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.533245    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:56:33.533451 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.533426    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:56:33.535587 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.535565    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"]
Oct 30 04:56:33.537193 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.537171    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:33.576293 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.573965    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhzpt\" (UniqueName: \"kubernetes.io/projected/35684429-00ad-418e-b11b-a8f5b1c0caef-kube-api-access-qhzpt\") pod \"maas-api-revoke-sub-2e135cd6-v8sr7\" (UID: \"35684429-00ad-418e-b11b-a8f5b1c0caef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:33.576293 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.574165    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e135cd6-v8sr7\" (UID: \"35684429-00ad-418e-b11b-a8f5b1c0caef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:33.576293 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.574358    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e135cd6-v8sr7\" (UID: \"35684429-00ad-418e-b11b-a8f5b1c0caef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:33.677217 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.677122    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e135cd6-v8sr7\" (UID: \"35684429-00ad-418e-b11b-a8f5b1c0caef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:33.678310 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.677550    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e135cd6-v8sr7\" (UID: \"35684429-00ad-418e-b11b-a8f5b1c0caef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:33.678310 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.677756    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhzpt\" (UniqueName: \"kubernetes.io/projected/35684429-00ad-418e-b11b-a8f5b1c0caef-kube-api-access-qhzpt\") pod \"maas-api-revoke-sub-2e135cd6-v8sr7\" (UID: \"35684429-00ad-418e-b11b-a8f5b1c0caef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:33.679595 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.679564    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e135cd6-v8sr7\" (UID: \"35684429-00ad-418e-b11b-a8f5b1c0caef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:33.707200 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.702983    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e135cd6-v8sr7\" (UID: \"35684429-00ad-418e-b11b-a8f5b1c0caef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:33.710897 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.710870    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhzpt\" (UniqueName: \"kubernetes.io/projected/35684429-00ad-418e-b11b-a8f5b1c0caef-kube-api-access-qhzpt\") pod \"maas-api-revoke-sub-2e135cd6-v8sr7\" (UID: \"35684429-00ad-418e-b11b-a8f5b1c0caef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:33.835308 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.835228    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:33.981130 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:33.980608    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"]
Oct 30 04:56:33.982297 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:56:33.982264    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f87a78_872f_4019_a991_ac26e9f90cef.slice/crio-d11e54e32b472cd108008566c6b2d6e56fd9ae5dc97c30070b2648adbafb4215 WatchSource:0}: Error finding container d11e54e32b472cd108008566c6b2d6e56fd9ae5dc97c30070b2648adbafb4215: Status 404 returned error can't find the container with id d11e54e32b472cd108008566c6b2d6e56fd9ae5dc97c30070b2648adbafb4215
Oct 30 04:56:34.021673 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.021647    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258" event={"ID":"62f87a78-872f-4019-a991-ac26e9f90cef","Type":"ContainerStarted","Data":"d11e54e32b472cd108008566c6b2d6e56fd9ae5dc97c30070b2648adbafb4215"}
Oct 30 04:56:34.189666 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:56:34.189631    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35684429_00ad_418e_b11b_a8f5b1c0caef.slice/crio-cabf508f8856c617f28695c3e504ad0d0a5880a8cafeef50c6494937343902f9 WatchSource:0}: Error finding container cabf508f8856c617f28695c3e504ad0d0a5880a8cafeef50c6494937343902f9: Status 404 returned error can't find the container with id cabf508f8856c617f28695c3e504ad0d0a5880a8cafeef50c6494937343902f9
Oct 30 04:56:34.192725 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.192698    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"]
Oct 30 04:56:34.347418 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.347329    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"]
Oct 30 04:56:34.355476 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.355440    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.370424 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.370395    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"]
Oct 30 04:56:34.500116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.495137    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.500116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.495202    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.500116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.495254    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bjc\" (UniqueName: \"kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-kube-api-access-v4bjc\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.500116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.495422    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.596409 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.596344    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.596409 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.596394    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.596696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.596454    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bjc\" (UniqueName: \"kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-kube-api-access-v4bjc\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.596696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.596641    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.597671 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.597608    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.597671 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.597608    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.599552 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.599527    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.621051 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.621022    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bjc\" (UniqueName: \"kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-kube-api-access-v4bjc\") pod \"payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") " pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:34.674490 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:34.674467    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:35.000537 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:56:35.000504    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3938dae4_a239_4a1f_a6a3_7a6461e8eb56.slice/crio-55182894275cbce57c8178a55dd3ef0ccf135552ee4ecb4656aa7156773bf3fb WatchSource:0}: Error finding container 55182894275cbce57c8178a55dd3ef0ccf135552ee4ecb4656aa7156773bf3fb: Status 404 returned error can't find the container with id 55182894275cbce57c8178a55dd3ef0ccf135552ee4ecb4656aa7156773bf3fb
Oct 30 04:56:35.000944 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:35.000922    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"]
Oct 30 04:56:35.031522 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:35.031488    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7" event={"ID":"35684429-00ad-418e-b11b-a8f5b1c0caef","Type":"ContainerStarted","Data":"c5b0af732d8fcb8ec11e56cec8f9f93a0cce88d1f4d4fda917e91491899587af"}
Oct 30 04:56:35.031659 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:35.031551    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7" event={"ID":"35684429-00ad-418e-b11b-a8f5b1c0caef","Type":"ContainerStarted","Data":"cabf508f8856c617f28695c3e504ad0d0a5880a8cafeef50c6494937343902f9"}
Oct 30 04:56:35.032974 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:35.032946    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg" event={"ID":"3938dae4-a239-4a1f-a6a3-7a6461e8eb56","Type":"ContainerStarted","Data":"55182894275cbce57c8178a55dd3ef0ccf135552ee4ecb4656aa7156773bf3fb"}
Oct 30 04:56:35.035013 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:35.034985    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258" event={"ID":"62f87a78-872f-4019-a991-ac26e9f90cef","Type":"ContainerStarted","Data":"18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61"}
Oct 30 04:56:35.035149 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:35.035115    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:35.076261 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:35.076216    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7" podStartSLOduration=2.076202071 podStartE2EDuration="2.076202071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:35.07288527 +0000 UTC m=+3574.814308370" watchObservedRunningTime="2026-10-30 04:56:35.076202071 +0000 UTC m=+3574.817625162"
Oct 30 04:56:35.118922 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:35.118859    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258" podStartSLOduration=2.118842425 podStartE2EDuration="2.118842425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:35.115504497 +0000 UTC m=+3574.856927595" watchObservedRunningTime="2026-10-30 04:56:35.118842425 +0000 UTC m=+3574.860265521"
Oct 30 04:56:36.044819 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:36.044783    5072 generic.go:356] "Generic (PLEG): container finished" podID="35684429-00ad-418e-b11b-a8f5b1c0caef" containerID="c5b0af732d8fcb8ec11e56cec8f9f93a0cce88d1f4d4fda917e91491899587af" exitCode=0
Oct 30 04:56:36.045314 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:36.044879    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7" event={"ID":"35684429-00ad-418e-b11b-a8f5b1c0caef","Type":"ContainerDied","Data":"c5b0af732d8fcb8ec11e56cec8f9f93a0cce88d1f4d4fda917e91491899587af"}
Oct 30 04:56:36.046608 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:36.046582    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg" event={"ID":"3938dae4-a239-4a1f-a6a3-7a6461e8eb56","Type":"ContainerStarted","Data":"8691531fb791c43b3e68094e765251e79853009dc1eb6c3e70e44e986567c4c6"}
Oct 30 04:56:36.131726 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:36.131675    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg" podStartSLOduration=2.131662626 podStartE2EDuration="2.131662626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:36.128552985 +0000 UTC m=+3575.869976106" watchObservedRunningTime="2026-10-30 04:56:36.131662626 +0000 UTC m=+3575.873085722"
Oct 30 04:56:37.034812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.034781    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"]
Oct 30 04:56:37.046866 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.046832    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"]
Oct 30 04:56:37.054974 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.054939    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692dddcb-d64wg"]
Oct 30 04:56:37.061991 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.061960    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e45d13-ghvxq"]
Oct 30 04:56:37.063284 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.063260    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:37.232238 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.232211    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:37.405753 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.405675    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-openshift-ca\") pod \"35684429-00ad-418e-b11b-a8f5b1c0caef\" (UID: \"35684429-00ad-418e-b11b-a8f5b1c0caef\") "
Oct 30 04:56:37.405919 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.405799    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35684429-00ad-418e-b11b-a8f5b1c0caef-kube-api-access-qhzpt\" (UniqueName: \"kubernetes.io/projected/35684429-00ad-418e-b11b-a8f5b1c0caef-kube-api-access-qhzpt\") pod \"35684429-00ad-418e-b11b-a8f5b1c0caef\" (UID: \"35684429-00ad-418e-b11b-a8f5b1c0caef\") "
Oct 30 04:56:37.405919 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.405870    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-xks-ca\") pod \"35684429-00ad-418e-b11b-a8f5b1c0caef\" (UID: \"35684429-00ad-418e-b11b-a8f5b1c0caef\") "
Oct 30 04:56:37.406122 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.406091    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-openshift-ca" pod "35684429-00ad-418e-b11b-a8f5b1c0caef" (UID: "35684429-00ad-418e-b11b-a8f5b1c0caef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:37.406429 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.406408    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:37.408168 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.408143    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35684429-00ad-418e-b11b-a8f5b1c0caef-kube-api-access-qhzpt" pod "35684429-00ad-418e-b11b-a8f5b1c0caef" (UID: "35684429-00ad-418e-b11b-a8f5b1c0caef"). InnerVolumeSpecName "kube-api-access-qhzpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:37.408277 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.408210    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-xks-ca" pod "35684429-00ad-418e-b11b-a8f5b1c0caef" (UID: "35684429-00ad-418e-b11b-a8f5b1c0caef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:56:37.507359 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.507325    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhzpt\" (UniqueName: \"kubernetes.io/projected/35684429-00ad-418e-b11b-a8f5b1c0caef-kube-api-access-qhzpt\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:37.507359 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:37.507354    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35684429-00ad-418e-b11b-a8f5b1c0caef-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:38.072564 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:38.072531    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"
Oct 30 04:56:38.073034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:38.072556    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7" event={"ID":"35684429-00ad-418e-b11b-a8f5b1c0caef","Type":"ContainerDied","Data":"cabf508f8856c617f28695c3e504ad0d0a5880a8cafeef50c6494937343902f9"}
Oct 30 04:56:38.073034 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:38.072616    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cabf508f8856c617f28695c3e504ad0d0a5880a8cafeef50c6494937343902f9"
Oct 30 04:56:38.417579 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:38.415004    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91788bca-5e08-4ec9-8011-2162a3db043e" path="/var/lib/kubelet/pods/91788bca-5e08-4ec9-8011-2162a3db043e/volumes"
Oct 30 04:56:38.417579 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:38.415695    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f6c0f0-7776-488c-ba54-756479ac002b" path="/var/lib/kubelet/pods/c6f6c0f0-7776-488c-ba54-756479ac002b/volumes"
Oct 30 04:56:40.957424 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:40.957395    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"
Oct 30 04:56:40.957942 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:40.957516    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"
Oct 30 04:56:41.756128 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:56:41.755832    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:56:44.307137 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.307096    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"]
Oct 30 04:56:44.307978 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.307961    5072 cpu_manager.go:414] "containerMap: removing container" podUID="35684429-00ad-418e-b11b-a8f5b1c0caef" containerName="revoke-keys"
Oct 30 04:56:44.307978 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.307979    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35684429-00ad-418e-b11b-a8f5b1c0caef" containerName="revoke-keys"
Oct 30 04:56:44.308155 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.308147    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="35684429-00ad-418e-b11b-a8f5b1c0caef" containerName="revoke-keys"
Oct 30 04:56:44.313713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.313690    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:44.333126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.333055    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"]
Oct 30 04:56:44.340006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.339982    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:56:44.340006 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.340005    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 04:56:44.423800 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.423763    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb71983a-5hqgj\" (UID: \"b39aac01-debe-4cb2-81c4-90e41bf627b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:44.423960 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.423815    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb71983a-5hqgj\" (UID: \"b39aac01-debe-4cb2-81c4-90e41bf627b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:44.423960 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.423919    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gtz2\" (UniqueName: \"kubernetes.io/projected/b39aac01-debe-4cb2-81c4-90e41bf627b9-kube-api-access-8gtz2\") pod \"maas-api-revoke-sub-fb71983a-5hqgj\" (UID: \"b39aac01-debe-4cb2-81c4-90e41bf627b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:44.525269 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.525224    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gtz2\" (UniqueName: \"kubernetes.io/projected/b39aac01-debe-4cb2-81c4-90e41bf627b9-kube-api-access-8gtz2\") pod \"maas-api-revoke-sub-fb71983a-5hqgj\" (UID: \"b39aac01-debe-4cb2-81c4-90e41bf627b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:44.525484 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.525453    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb71983a-5hqgj\" (UID: \"b39aac01-debe-4cb2-81c4-90e41bf627b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:44.525635 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.525516    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb71983a-5hqgj\" (UID: \"b39aac01-debe-4cb2-81c4-90e41bf627b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:44.526361 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.526331    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb71983a-5hqgj\" (UID: \"b39aac01-debe-4cb2-81c4-90e41bf627b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:44.528103 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.528063    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb71983a-5hqgj\" (UID: \"b39aac01-debe-4cb2-81c4-90e41bf627b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:44.550369 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.550347    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gtz2\" (UniqueName: \"kubernetes.io/projected/b39aac01-debe-4cb2-81c4-90e41bf627b9-kube-api-access-8gtz2\") pod \"maas-api-revoke-sub-fb71983a-5hqgj\" (UID: \"b39aac01-debe-4cb2-81c4-90e41bf627b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:44.631559 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.631487    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:44.942447 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:56:44.942414    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39aac01_debe_4cb2_81c4_90e41bf627b9.slice/crio-27699e73a212b3d6c92ac212737f9917a6d9ffa4c6b96204562bdcc35e0f57a6 WatchSource:0}: Error finding container 27699e73a212b3d6c92ac212737f9917a6d9ffa4c6b96204562bdcc35e0f57a6: Status 404 returned error can't find the container with id 27699e73a212b3d6c92ac212737f9917a6d9ffa4c6b96204562bdcc35e0f57a6
Oct 30 04:56:44.945190 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:44.945093    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"]
Oct 30 04:56:45.199443 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:45.199360    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj" event={"ID":"b39aac01-debe-4cb2-81c4-90e41bf627b9","Type":"ContainerStarted","Data":"27699e73a212b3d6c92ac212737f9917a6d9ffa4c6b96204562bdcc35e0f57a6"}
Oct 30 04:56:46.049906 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:46.049878    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:56:46.130999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:46.130961    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"]
Oct 30 04:56:46.215974 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:46.215931    5072 generic.go:356] "Generic (PLEG): container finished" podID="b39aac01-debe-4cb2-81c4-90e41bf627b9" containerID="9924e1db6442003f228ed6e2a6b2c28fdab8d0bd7b7e6c3fa2c8bc4cfbbe773c" exitCode=0
Oct 30 04:56:46.216200 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:46.215974    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj" event={"ID":"b39aac01-debe-4cb2-81c4-90e41bf627b9","Type":"ContainerDied","Data":"9924e1db6442003f228ed6e2a6b2c28fdab8d0bd7b7e6c3fa2c8bc4cfbbe773c"}
Oct 30 04:56:47.398821 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:47.398795    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:47.471291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:47.471258    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39aac01-debe-4cb2-81c4-90e41bf627b9-kube-api-access-8gtz2\" (UniqueName: \"kubernetes.io/projected/b39aac01-debe-4cb2-81c4-90e41bf627b9-kube-api-access-8gtz2\") pod \"b39aac01-debe-4cb2-81c4-90e41bf627b9\" (UID: \"b39aac01-debe-4cb2-81c4-90e41bf627b9\") "
Oct 30 04:56:47.471451 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:47.471418    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-openshift-ca\") pod \"b39aac01-debe-4cb2-81c4-90e41bf627b9\" (UID: \"b39aac01-debe-4cb2-81c4-90e41bf627b9\") "
Oct 30 04:56:47.471599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:47.471571    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-xks-ca\") pod \"b39aac01-debe-4cb2-81c4-90e41bf627b9\" (UID: \"b39aac01-debe-4cb2-81c4-90e41bf627b9\") "
Oct 30 04:56:47.471899 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:47.471864    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-openshift-ca" pod "b39aac01-debe-4cb2-81c4-90e41bf627b9" (UID: "b39aac01-debe-4cb2-81c4-90e41bf627b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:47.472321 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:47.472295    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:47.474321 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:47.474293    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-xks-ca" pod "b39aac01-debe-4cb2-81c4-90e41bf627b9" (UID: "b39aac01-debe-4cb2-81c4-90e41bf627b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:56:47.474415 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:47.474360    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39aac01-debe-4cb2-81c4-90e41bf627b9-kube-api-access-8gtz2" pod "b39aac01-debe-4cb2-81c4-90e41bf627b9" (UID: "b39aac01-debe-4cb2-81c4-90e41bf627b9"). InnerVolumeSpecName "kube-api-access-8gtz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:47.573680 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:47.573646    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39aac01-debe-4cb2-81c4-90e41bf627b9-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:47.573680 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:47.573680    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gtz2\" (UniqueName: \"kubernetes.io/projected/b39aac01-debe-4cb2-81c4-90e41bf627b9-kube-api-access-8gtz2\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:48.034845 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:48.034810    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"]
Oct 30 04:56:48.057008 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:48.056971    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"]
Oct 30 04:56:48.057285 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:48.057266    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcae8d78-pzjxq"]
Oct 30 04:56:48.065605 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:48.065575    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac36228-bf88m"]
Oct 30 04:56:48.076614 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:48.076591    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:56:48.167643 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:48.167611    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"]
Oct 30 04:56:48.234450 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:48.234414    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj" event={"ID":"b39aac01-debe-4cb2-81c4-90e41bf627b9","Type":"ContainerDied","Data":"27699e73a212b3d6c92ac212737f9917a6d9ffa4c6b96204562bdcc35e0f57a6"}
Oct 30 04:56:48.234648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:48.234467    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27699e73a212b3d6c92ac212737f9917a6d9ffa4c6b96204562bdcc35e0f57a6"
Oct 30 04:56:48.234648 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:48.234519    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"
Oct 30 04:56:48.410672 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:48.410601    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="149f3d6b-e599-41ed-be75-c3311ac3c061" path="/var/lib/kubelet/pods/149f3d6b-e599-41ed-be75-c3311ac3c061/volumes"
Oct 30 04:56:48.411289 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:48.411264    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568b2434-830f-4e8f-9f96-42c8cf3f6ffa" path="/var/lib/kubelet/pods/568b2434-830f-4e8f-9f96-42c8cf3f6ffa/volumes"
Oct 30 04:56:51.131711 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.131667    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf" podUID="bbbe6653-c77e-4408-b6f0-bd16127b2d6b" containerName="payload-processing" containerID="cri-o://21a4fbcc2cac20a05a7c1e529a956dcd7434dbcc8ba9205a23f750bdffc72cf2" gracePeriod=25
Oct 30 04:56:51.264203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.264172    5072 generic.go:356] "Generic (PLEG): container finished" podID="bbbe6653-c77e-4408-b6f0-bd16127b2d6b" containerID="21a4fbcc2cac20a05a7c1e529a956dcd7434dbcc8ba9205a23f750bdffc72cf2" exitCode=0
Oct 30 04:56:51.264366 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.264223    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf" event={"ID":"bbbe6653-c77e-4408-b6f0-bd16127b2d6b","Type":"ContainerDied","Data":"21a4fbcc2cac20a05a7c1e529a956dcd7434dbcc8ba9205a23f750bdffc72cf2"}
Oct 30 04:56:51.423230 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.423204    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:56:51.531289 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.531252    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-provider-credentials\") pod \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") "
Oct 30 04:56:51.531503 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.531334    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-plugins-config-volume\") pod \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") "
Oct 30 04:56:51.531503 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.531400    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-routing-overlay-volume\") pod \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") "
Oct 30 04:56:51.531696 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.531680    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-kube-api-access-pjlbt\" (UniqueName: \"kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-kube-api-access-pjlbt\") pod \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\" (UID: \"bbbe6653-c77e-4408-b6f0-bd16127b2d6b\") "
Oct 30 04:56:51.531777 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.531750    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-routing-overlay-volume" pod "bbbe6653-c77e-4408-b6f0-bd16127b2d6b" (UID: "bbbe6653-c77e-4408-b6f0-bd16127b2d6b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:51.531838 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.531770    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-plugins-config-volume" pod "bbbe6653-c77e-4408-b6f0-bd16127b2d6b" (UID: "bbbe6653-c77e-4408-b6f0-bd16127b2d6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:51.532384 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.532359    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:51.532384 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.532384    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:51.533941 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.533915    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-provider-credentials" pod "bbbe6653-c77e-4408-b6f0-bd16127b2d6b" (UID: "bbbe6653-c77e-4408-b6f0-bd16127b2d6b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:51.534134 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.534054    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-kube-api-access-pjlbt" pod "bbbe6653-c77e-4408-b6f0-bd16127b2d6b" (UID: "bbbe6653-c77e-4408-b6f0-bd16127b2d6b"). InnerVolumeSpecName "kube-api-access-pjlbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:51.633536 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.633507    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjlbt\" (UniqueName: \"kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-kube-api-access-pjlbt\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:51.633536 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:51.633536    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbe6653-c77e-4408-b6f0-bd16127b2d6b-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:52.277170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:52.277134    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"
Oct 30 04:56:52.277170 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:52.277144    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf" event={"ID":"bbbe6653-c77e-4408-b6f0-bd16127b2d6b","Type":"ContainerDied","Data":"e5cd3368fe0ac3bdb3ee4b8690bad32fa09883bf0ddfb653fa03ef9b3a5b5db8"}
Oct 30 04:56:52.277724 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:52.277222    5072 scope.go:122] "RemoveContainer" containerID="21a4fbcc2cac20a05a7c1e529a956dcd7434dbcc8ba9205a23f750bdffc72cf2"
Oct 30 04:56:52.335004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:52.334976    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"]
Oct 30 04:56:52.339220 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:52.339195    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-779fs8kmf"]
Oct 30 04:56:52.411069 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:52.411042    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbe6653-c77e-4408-b6f0-bd16127b2d6b" path="/var/lib/kubelet/pods/bbbe6653-c77e-4408-b6f0-bd16127b2d6b/volumes"
Oct 30 04:56:53.168817 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.168772    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6" podUID="67c75bb2-f1e9-4ef8-b536-73d8f299289a" containerName="payload-processing" containerID="cri-o://b70b9aade521c4190cc337bac4824dea17d72eedb9f9e59ae66af7553a60b88f" gracePeriod=25
Oct 30 04:56:53.288279 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.288251    5072 generic.go:356] "Generic (PLEG): container finished" podID="67c75bb2-f1e9-4ef8-b536-73d8f299289a" containerID="b70b9aade521c4190cc337bac4824dea17d72eedb9f9e59ae66af7553a60b88f" exitCode=0
Oct 30 04:56:53.288711 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.288317    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6" event={"ID":"67c75bb2-f1e9-4ef8-b536-73d8f299289a","Type":"ContainerDied","Data":"b70b9aade521c4190cc337bac4824dea17d72eedb9f9e59ae66af7553a60b88f"}
Oct 30 04:56:53.451847 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.451824    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:56:53.560747 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.560716    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-plugins-config-volume\") pod \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") "
Oct 30 04:56:53.560929 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.560779    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-routing-overlay-volume\") pod \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") "
Oct 30 04:56:53.560929 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.560806    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-kube-api-access-l4f6x\" (UniqueName: \"kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-kube-api-access-l4f6x\") pod \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") "
Oct 30 04:56:53.561045 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.560915    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-provider-credentials\") pod \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\" (UID: \"67c75bb2-f1e9-4ef8-b536-73d8f299289a\") "
Oct 30 04:56:53.561203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.561177    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-routing-overlay-volume" pod "67c75bb2-f1e9-4ef8-b536-73d8f299289a" (UID: "67c75bb2-f1e9-4ef8-b536-73d8f299289a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:53.561293 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.561182    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-plugins-config-volume" pod "67c75bb2-f1e9-4ef8-b536-73d8f299289a" (UID: "67c75bb2-f1e9-4ef8-b536-73d8f299289a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:56:53.561707 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.561678    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:53.561838 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.561710    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c75bb2-f1e9-4ef8-b536-73d8f299289a-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:53.563118 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.563095    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-kube-api-access-l4f6x" pod "67c75bb2-f1e9-4ef8-b536-73d8f299289a" (UID: "67c75bb2-f1e9-4ef8-b536-73d8f299289a"). InnerVolumeSpecName "kube-api-access-l4f6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:53.563672 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.563650    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-provider-credentials" pod "67c75bb2-f1e9-4ef8-b536-73d8f299289a" (UID: "67c75bb2-f1e9-4ef8-b536-73d8f299289a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:53.662729 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.662685    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4f6x\" (UniqueName: \"kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-kube-api-access-l4f6x\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:53.662729 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:53.662719    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c75bb2-f1e9-4ef8-b536-73d8f299289a-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:56:54.297833 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:54.297801    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"
Oct 30 04:56:54.298330 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:54.297832    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6" event={"ID":"67c75bb2-f1e9-4ef8-b536-73d8f299289a","Type":"ContainerDied","Data":"0bc703caa31c74cee37876e5060cc41b0a298c84e076be35d8ff42fa928263db"}
Oct 30 04:56:54.298330 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:54.297900    5072 scope.go:122] "RemoveContainer" containerID="b70b9aade521c4190cc337bac4824dea17d72eedb9f9e59ae66af7553a60b88f"
Oct 30 04:56:54.358936 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:54.358899    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"]
Oct 30 04:56:54.373295 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:54.373259    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-7f6bvgmx6"]
Oct 30 04:56:54.412981 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:54.412942    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c75bb2-f1e9-4ef8-b536-73d8f299289a" path="/var/lib/kubelet/pods/67c75bb2-f1e9-4ef8-b536-73d8f299289a/volumes"
Oct 30 04:56:56.119645 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:56.119611    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"]
Oct 30 04:56:57.027531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:57.027501    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"]
Oct 30 04:56:59.602791 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.602755    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"]
Oct 30 04:56:59.603657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.603637    5072 cpu_manager.go:414] "containerMap: removing container" podUID="bbbe6653-c77e-4408-b6f0-bd16127b2d6b" containerName="payload-processing"
Oct 30 04:56:59.603657 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.603658    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbe6653-c77e-4408-b6f0-bd16127b2d6b" containerName="payload-processing"
Oct 30 04:56:59.603770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.603683    5072 cpu_manager.go:414] "containerMap: removing container" podUID="67c75bb2-f1e9-4ef8-b536-73d8f299289a" containerName="payload-processing"
Oct 30 04:56:59.603770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.603688    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c75bb2-f1e9-4ef8-b536-73d8f299289a" containerName="payload-processing"
Oct 30 04:56:59.603770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.603703    5072 cpu_manager.go:414] "containerMap: removing container" podUID="b39aac01-debe-4cb2-81c4-90e41bf627b9" containerName="revoke-keys"
Oct 30 04:56:59.603770 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.603709    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39aac01-debe-4cb2-81c4-90e41bf627b9" containerName="revoke-keys"
Oct 30 04:56:59.603891 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.603842    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbe6653-c77e-4408-b6f0-bd16127b2d6b" containerName="payload-processing"
Oct 30 04:56:59.603891 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.603853    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c75bb2-f1e9-4ef8-b536-73d8f299289a" containerName="payload-processing"
Oct 30 04:56:59.603891 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.603863    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39aac01-debe-4cb2-81c4-90e41bf627b9" containerName="revoke-keys"
Oct 30 04:56:59.609621 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.609598    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.629406 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.629378    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"]
Oct 30 04:56:59.636599 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.636566    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-82bec5\"/\"default-dockercfg-96ksf\""
Oct 30 04:56:59.636713 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.636632    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-82bec5\"/\"test-model-82bec5-kserve-self-signed-certs\""
Oct 30 04:56:59.639248 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.639226    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-82bec5\"/\"openshift-service-ca.crt\""
Oct 30 04:56:59.639507 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.639488    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-82bec5\"/\"kube-root-ca.crt\""
Oct 30 04:56:59.679656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.679621    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-home\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.679835 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.679661    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tmp-dir\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.679835 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.679810    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-dshm\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.679974 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.679943    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-model-cache\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.680040 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.680008    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-672x8\" (UniqueName: \"kubernetes.io/projected/092d6318-b45b-40d4-b3ec-7b2ad2769e04-kube-api-access-672x8\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.680145 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.680099    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tls-certs\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.781214 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.781174    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-home\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.781388 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.781230    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tmp-dir\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.781388 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.781378    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-dshm\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.781474 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.781445    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-model-cache\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.781515 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.781502    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672x8\" (UniqueName: \"kubernetes.io/projected/092d6318-b45b-40d4-b3ec-7b2ad2769e04-kube-api-access-672x8\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.781582 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.781571    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tls-certs\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.781669 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.781648    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-home\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.781923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.781759    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tmp-dir\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.781923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.781861    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-model-cache\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.784022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.783999    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-dshm\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.784704 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.784684    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tls-certs\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.805661 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.805635    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-672x8\" (UniqueName: \"kubernetes.io/projected/092d6318-b45b-40d4-b3ec-7b2ad2769e04-kube-api-access-672x8\") pod \"test-model-82bec5-kserve-56f6b87d77-vhfbh\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") " pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:56:59.937154 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:56:59.933363    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:57:00.256962 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:57:00.256935    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod092d6318_b45b_40d4_b3ec_7b2ad2769e04.slice/crio-bc0595e722c726d621b60b38978718e06402ceba02d2c0c75f92e54a68266eab WatchSource:0}: Error finding container bc0595e722c726d621b60b38978718e06402ceba02d2c0c75f92e54a68266eab: Status 404 returned error can't find the container with id bc0595e722c726d621b60b38978718e06402ceba02d2c0c75f92e54a68266eab
Oct 30 04:57:00.257296 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:00.257275    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"]
Oct 30 04:57:00.357126 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:00.357056    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh" event={"ID":"092d6318-b45b-40d4-b3ec-7b2ad2769e04","Type":"ContainerStarted","Data":"51aa9cd35ea1d353aa9c13384ad7fb15dafebab3b456d082fa015d512bbd1da0"}
Oct 30 04:57:00.357260 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:00.357139    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh" event={"ID":"092d6318-b45b-40d4-b3ec-7b2ad2769e04","Type":"ContainerStarted","Data":"bc0595e722c726d621b60b38978718e06402ceba02d2c0c75f92e54a68266eab"}
Oct 30 04:57:00.402965 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:00.402915    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh" podStartSLOduration=1.402901661 podStartE2EDuration="1.402901661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:00.399386182 +0000 UTC m=+3600.140809279" watchObservedRunningTime="2026-10-30 04:57:00.402901661 +0000 UTC m=+3600.144324758"
Oct 30 04:57:00.932598 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:00.932568    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:57:00.933533 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:00.933507    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:57:00.959127 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:00.959103    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 04:57:01.120115 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.120035    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg" podUID="3938dae4-a239-4a1f-a6a3-7a6461e8eb56" containerName="payload-processing" containerID="cri-o://8691531fb791c43b3e68094e765251e79853009dc1eb6c3e70e44e986567c4c6" gracePeriod=25
Oct 30 04:57:01.370364 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.370322    5072 generic.go:356] "Generic (PLEG): container finished" podID="3938dae4-a239-4a1f-a6a3-7a6461e8eb56" containerID="8691531fb791c43b3e68094e765251e79853009dc1eb6c3e70e44e986567c4c6" exitCode=0
Oct 30 04:57:01.370632 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.370602    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg" event={"ID":"3938dae4-a239-4a1f-a6a3-7a6461e8eb56","Type":"ContainerDied","Data":"8691531fb791c43b3e68094e765251e79853009dc1eb6c3e70e44e986567c4c6"}
Oct 30 04:57:01.442602 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.442573    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:57:01.503609 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.503506    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-provider-credentials\") pod \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") "
Oct 30 04:57:01.503795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.503628    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-plugins-config-volume\") pod \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") "
Oct 30 04:57:01.503795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.503654    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-kube-api-access-v4bjc\" (UniqueName: \"kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-kube-api-access-v4bjc\") pod \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") "
Oct 30 04:57:01.503795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.503697    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-routing-overlay-volume\") pod \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\" (UID: \"3938dae4-a239-4a1f-a6a3-7a6461e8eb56\") "
Oct 30 04:57:01.506683 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.504807    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-routing-overlay-volume" pod "3938dae4-a239-4a1f-a6a3-7a6461e8eb56" (UID: "3938dae4-a239-4a1f-a6a3-7a6461e8eb56"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:57:01.506683 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.505371    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-plugins-config-volume" pod "3938dae4-a239-4a1f-a6a3-7a6461e8eb56" (UID: "3938dae4-a239-4a1f-a6a3-7a6461e8eb56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:57:01.507604 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.507583    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-kube-api-access-v4bjc" pod "3938dae4-a239-4a1f-a6a3-7a6461e8eb56" (UID: "3938dae4-a239-4a1f-a6a3-7a6461e8eb56"). InnerVolumeSpecName "kube-api-access-v4bjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:57:01.507952 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.507926    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-provider-credentials" pod "3938dae4-a239-4a1f-a6a3-7a6461e8eb56" (UID: "3938dae4-a239-4a1f-a6a3-7a6461e8eb56"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:57:01.605762 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.605733    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:57:01.605762 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.605761    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:57:01.605941 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.605773    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4bjc\" (UniqueName: \"kubernetes.io/projected/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-kube-api-access-v4bjc\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:57:01.605941 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:01.605781    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3938dae4-a239-4a1f-a6a3-7a6461e8eb56-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:57:02.028315 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.028269    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258" podUID="62f87a78-872f-4019-a991-ac26e9f90cef" containerName="payload-processing" containerID="cri-o://18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61" gracePeriod=25
Oct 30 04:57:02.032375 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.032350    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"]
Oct 30 04:57:02.036633 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.036611    5072 scope.go:122] "RemoveContainer" containerID="6f24ea536ae585b7135d8cc5574749204fd6403f07684fe2e8626157fd6390fa"
Oct 30 04:57:02.045897 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.045875    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4859cfc6-sxpj8"]
Oct 30 04:57:02.052185 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.052167    5072 scope.go:122] "RemoveContainer" containerID="cd516aa05d94ea9e5f571b25053d550b60e01b0affcf762522e934a854dfa72c"
Oct 30 04:57:02.143538 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.143307    5072 scope.go:122] "RemoveContainer" containerID="9d298192a58bd1a54a64b4658dd3f4f7ab9d41a2e1d9b86c7bb1daa417defe42"
Oct 30 04:57:02.344927 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.344901    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:57:02.382032 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.382004    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"
Oct 30 04:57:02.382291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.382257    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg" event={"ID":"3938dae4-a239-4a1f-a6a3-7a6461e8eb56","Type":"ContainerDied","Data":"55182894275cbce57c8178a55dd3ef0ccf135552ee4ecb4656aa7156773bf3fb"}
Oct 30 04:57:02.382410 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.382361    5072 scope.go:122] "RemoveContainer" containerID="8691531fb791c43b3e68094e765251e79853009dc1eb6c3e70e44e986567c4c6"
Oct 30 04:57:02.385803 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.385770    5072 generic.go:356] "Generic (PLEG): container finished" podID="62f87a78-872f-4019-a991-ac26e9f90cef" containerID="18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61" exitCode=0
Oct 30 04:57:02.386052 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.385896    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258" event={"ID":"62f87a78-872f-4019-a991-ac26e9f90cef","Type":"ContainerDied","Data":"18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61"}
Oct 30 04:57:02.386052 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.385942    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258" event={"ID":"62f87a78-872f-4019-a991-ac26e9f90cef","Type":"ContainerDied","Data":"d11e54e32b472cd108008566c6b2d6e56fd9ae5dc97c30070b2648adbafb4215"}
Oct 30 04:57:02.386052 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.385988    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"
Oct 30 04:57:02.405792 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.405765    5072 scope.go:122] "RemoveContainer" containerID="18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61"
Oct 30 04:57:02.419155 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.419127    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-routing-overlay-volume\") pod \"62f87a78-872f-4019-a991-ac26e9f90cef\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") "
Oct 30 04:57:02.419292 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.419177    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-plugins-config-volume\") pod \"62f87a78-872f-4019-a991-ac26e9f90cef\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") "
Oct 30 04:57:02.419359 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.419292    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-kube-api-access-cff9n\" (UniqueName: \"kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-kube-api-access-cff9n\") pod \"62f87a78-872f-4019-a991-ac26e9f90cef\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") "
Oct 30 04:57:02.419512 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.419486    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-routing-overlay-volume" pod "62f87a78-872f-4019-a991-ac26e9f90cef" (UID: "62f87a78-872f-4019-a991-ac26e9f90cef"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:57:02.419656 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.419631    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-plugins-config-volume" pod "62f87a78-872f-4019-a991-ac26e9f90cef" (UID: "62f87a78-872f-4019-a991-ac26e9f90cef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:57:02.420087 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.420047    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-provider-credentials\") pod \"62f87a78-872f-4019-a991-ac26e9f90cef\" (UID: \"62f87a78-872f-4019-a991-ac26e9f90cef\") "
Oct 30 04:57:02.420589 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.420570    5072 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-routing-overlay-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:57:02.420589 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.420591    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f87a78-872f-4019-a991-ac26e9f90cef-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:57:02.422606 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.422570    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-kube-api-access-cff9n" pod "62f87a78-872f-4019-a991-ac26e9f90cef" (UID: "62f87a78-872f-4019-a991-ac26e9f90cef"). InnerVolumeSpecName "kube-api-access-cff9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:57:02.425575 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.425553    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2907e444-12e2-47cd-b6d8-b5b4183ec275" path="/var/lib/kubelet/pods/2907e444-12e2-47cd-b6d8-b5b4183ec275/volumes"
Oct 30 04:57:02.433923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.433898    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-provider-credentials" pod "62f87a78-872f-4019-a991-ac26e9f90cef" (UID: "62f87a78-872f-4019-a991-ac26e9f90cef"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:57:02.450684 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.450660    5072 scope.go:122] "RemoveContainer" containerID="18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61"
Oct 30 04:57:02.451087 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:57:02.451049    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61\": container with ID starting with 18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61 not found: ID does not exist" containerID="18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61"
Oct 30 04:57:02.451162 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.451101    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61"} err="failed to get container status \"18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61\": rpc error: code = NotFound desc = could not find container \"18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61\": container with ID starting with 18a892af39231a096f72c64d961ea1aea2d1f076bb2b7fa20a5e544280266e61 not found: ID does not exist"
Oct 30 04:57:02.474871 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.474839    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"]
Oct 30 04:57:02.479161 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.479137    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ed8266/payload-processing-external-model-e2e-extmod-a-ed8266-ccb7849bg"]
Oct 30 04:57:02.521864 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.521833    5072 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-provider-credentials\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:57:02.521864 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.521862    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cff9n\" (UniqueName: \"kubernetes.io/projected/62f87a78-872f-4019-a991-ac26e9f90cef-kube-api-access-cff9n\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 04:57:02.740300 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.740270    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"]
Oct 30 04:57:02.744303 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:02.744280    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-84482b258"]
Oct 30 04:57:04.426231 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:04.426191    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3938dae4-a239-4a1f-a6a3-7a6461e8eb56" path="/var/lib/kubelet/pods/3938dae4-a239-4a1f-a6a3-7a6461e8eb56/volumes"
Oct 30 04:57:04.426826 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:04.426807    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f87a78-872f-4019-a991-ac26e9f90cef" path="/var/lib/kubelet/pods/62f87a78-872f-4019-a991-ac26e9f90cef/volumes"
Oct 30 04:57:09.934163 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:09.934126    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:57:09.949839 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:09.949806    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:57:10.205911 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.205880    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"]
Oct 30 04:57:10.206775 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.206755    5072 cpu_manager.go:414] "containerMap: removing container" podUID="3938dae4-a239-4a1f-a6a3-7a6461e8eb56" containerName="payload-processing"
Oct 30 04:57:10.206821 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.206777    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3938dae4-a239-4a1f-a6a3-7a6461e8eb56" containerName="payload-processing"
Oct 30 04:57:10.206870 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.206822    5072 cpu_manager.go:414] "containerMap: removing container" podUID="62f87a78-872f-4019-a991-ac26e9f90cef" containerName="payload-processing"
Oct 30 04:57:10.206870 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.206827    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f87a78-872f-4019-a991-ac26e9f90cef" containerName="payload-processing"
Oct 30 04:57:10.206975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.206947    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f87a78-872f-4019-a991-ac26e9f90cef" containerName="payload-processing"
Oct 30 04:57:10.206975 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.206961    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3938dae4-a239-4a1f-a6a3-7a6461e8eb56" containerName="payload-processing"
Oct 30 04:57:10.212506 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.212486    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"
Oct 30 04:57:10.258669 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.258636    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9f158a\""
Oct 30 04:57:10.275455 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.275417    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9f158a-dockercfg-nb4nq\""
Oct 30 04:57:10.282574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.282542    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"]
Oct 30 04:57:10.307788 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.305967    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"]
Oct 30 04:57:10.314476 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.314442    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"
Oct 30 04:57:10.337060 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.337026    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"]
Oct 30 04:57:10.439986 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.439943    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttrbd\" (UniqueName: \"kubernetes.io/projected/9ad68684-c8f0-4656-9bde-67293af60323-kube-api-access-ttrbd\") pod \"payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85\" (UID: \"9ad68684-c8f0-4656-9bde-67293af60323\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"
Oct 30 04:57:10.440203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.440111    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad68684-c8f0-4656-9bde-67293af60323-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85\" (UID: \"9ad68684-c8f0-4656-9bde-67293af60323\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"
Oct 30 04:57:10.514477 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.514393    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv\" (UID: \"2e3a563c-81d1-41a3-9c9a-ffe52517fd03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"
Oct 30 04:57:10.514849 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.514826    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blp4w\" (UniqueName: \"kubernetes.io/projected/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-kube-api-access-blp4w\") pod \"payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv\" (UID: \"2e3a563c-81d1-41a3-9c9a-ffe52517fd03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"
Oct 30 04:57:10.549438 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.549405    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 04:57:10.618092 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.618030    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad68684-c8f0-4656-9bde-67293af60323-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85\" (UID: \"9ad68684-c8f0-4656-9bde-67293af60323\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"
Oct 30 04:57:10.618291 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.618145    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv\" (UID: \"2e3a563c-81d1-41a3-9c9a-ffe52517fd03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"
Oct 30 04:57:10.618364 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.618328    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blp4w\" (UniqueName: \"kubernetes.io/projected/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-kube-api-access-blp4w\") pod \"payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv\" (UID: \"2e3a563c-81d1-41a3-9c9a-ffe52517fd03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"
Oct 30 04:57:10.618737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.618637    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttrbd\" (UniqueName: \"kubernetes.io/projected/9ad68684-c8f0-4656-9bde-67293af60323-kube-api-access-ttrbd\") pod \"payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85\" (UID: \"9ad68684-c8f0-4656-9bde-67293af60323\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"
Oct 30 04:57:10.619004 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.618982    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv\" (UID: \"2e3a563c-81d1-41a3-9c9a-ffe52517fd03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"
Oct 30 04:57:10.619106 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.619018    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad68684-c8f0-4656-9bde-67293af60323-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85\" (UID: \"9ad68684-c8f0-4656-9bde-67293af60323\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"
Oct 30 04:57:10.649120 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.649044    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttrbd\" (UniqueName: \"kubernetes.io/projected/9ad68684-c8f0-4656-9bde-67293af60323-kube-api-access-ttrbd\") pod \"payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85\" (UID: \"9ad68684-c8f0-4656-9bde-67293af60323\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"
Oct 30 04:57:10.651769 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.651742    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blp4w\" (UniqueName: \"kubernetes.io/projected/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-kube-api-access-blp4w\") pod \"payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv\" (UID: \"2e3a563c-81d1-41a3-9c9a-ffe52517fd03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"
Oct 30 04:57:10.827381 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.827285    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"
Oct 30 04:57:10.868356 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.868322    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"]
Oct 30 04:57:10.874585 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.874561    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:10.896809 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.896764    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"]
Oct 30 04:57:10.939825 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.939454    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"
Oct 30 04:57:10.975914 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:10.975883    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5f1508-openshift-default-dockercfg-4jlkh\""
Oct 30 04:57:11.028203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.027700    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-podinfo\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.028203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.027777    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvqvz\" (UniqueName: \"kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-kube-api-access-lvqvz\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.028203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.027808    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-data\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.028203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.027915    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-socket\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.028203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.027944    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-token\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.028203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.028006    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istiod-ca-cert\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.028203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.028135    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-credential-socket\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.028203 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.028165    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-certs\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.028799 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.028329    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-envoy\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.129674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.129590    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-podinfo\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.129674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.129650    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvqvz\" (UniqueName: \"kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-kube-api-access-lvqvz\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.129674 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.129669    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-data\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.129894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.129743    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-socket\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.129894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.129764    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-token\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.129894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.129808    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istiod-ca-cert\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.129894 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.129890    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-credential-socket\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.130035 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.129912    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-certs\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.130035 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.129941    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-envoy\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.131735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.130785    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-socket\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.131735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.131050    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-data\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.131735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.131262    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-credential-socket\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.131735 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.131413    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-certs\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.132025 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.131832    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istiod-ca-cert\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.133068 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.133041    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-envoy\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.134778 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.134753    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-podinfo\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.155938 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.155904    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-token\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.156271 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.156250    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvqvz\" (UniqueName: \"kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-kube-api-access-lvqvz\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.161774 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.161747    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"]
Oct 30 04:57:11.167102 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:57:11.167057    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3a563c_81d1_41a3_9c9a_ffe52517fd03.slice/crio-a7a8bebabc24c1987d0569e3a7e68638a7ba8020a667fc6ecfcdf5570866577f WatchSource:0}: Error finding container a7a8bebabc24c1987d0569e3a7e68638a7ba8020a667fc6ecfcdf5570866577f: Status 404 returned error can't find the container with id a7a8bebabc24c1987d0569e3a7e68638a7ba8020a667fc6ecfcdf5570866577f
Oct 30 04:57:11.203597 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.203569    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:11.293333 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:57:11.292387    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad68684_c8f0_4656_9bde_67293af60323.slice/crio-f7134d586bef8c887ea5382f48c47c8369e5af1f20af600b25017696d2abd339 WatchSource:0}: Error finding container f7134d586bef8c887ea5382f48c47c8369e5af1f20af600b25017696d2abd339: Status 404 returned error can't find the container with id f7134d586bef8c887ea5382f48c47c8369e5af1f20af600b25017696d2abd339
Oct 30 04:57:11.293901 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.293870    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"]
Oct 30 04:57:11.503364 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.503303    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85" event={"ID":"9ad68684-c8f0-4656-9bde-67293af60323","Type":"ContainerStarted","Data":"35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897"}
Oct 30 04:57:11.503559 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.503382    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"
Oct 30 04:57:11.503559 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.503404    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85" event={"ID":"9ad68684-c8f0-4656-9bde-67293af60323","Type":"ContainerStarted","Data":"f7134d586bef8c887ea5382f48c47c8369e5af1f20af600b25017696d2abd339"}
Oct 30 04:57:11.505520 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.505456    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv" event={"ID":"2e3a563c-81d1-41a3-9c9a-ffe52517fd03","Type":"ContainerStarted","Data":"3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98"}
Oct 30 04:57:11.505679 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.505552    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv" event={"ID":"2e3a563c-81d1-41a3-9c9a-ffe52517fd03","Type":"ContainerStarted","Data":"a7a8bebabc24c1987d0569e3a7e68638a7ba8020a667fc6ecfcdf5570866577f"}
Oct 30 04:57:11.505757 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.505736    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"
Oct 30 04:57:11.527563 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:57:11.527539    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d290bd_4133_45ce_9816_a7b94e8d89a4.slice/crio-7ff76620fb44fa946a18dc1881750d0857b667ee694f93ded61b6afd99e04007 WatchSource:0}: Error finding container 7ff76620fb44fa946a18dc1881750d0857b667ee694f93ded61b6afd99e04007: Status 404 returned error can't find the container with id 7ff76620fb44fa946a18dc1881750d0857b667ee694f93ded61b6afd99e04007
Oct 30 04:57:11.528531 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.528504    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"]
Oct 30 04:57:11.530413 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.530380    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:57:11.530519 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.530454    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:57:11.530519 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.530502    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:57:11.547759 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.547709    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85" podStartSLOduration=1.547691801 podStartE2EDuration="1.547691801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:11.544734338 +0000 UTC m=+3611.286157472" watchObservedRunningTime="2026-10-30 04:57:11.547691801 +0000 UTC m=+3611.289114900"
Oct 30 04:57:11.590737 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:11.590670    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv" podStartSLOduration=1.590654035 podStartE2EDuration="1.590654035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:11.590162603 +0000 UTC m=+3611.331585699" watchObservedRunningTime="2026-10-30 04:57:11.590654035 +0000 UTC m=+3611.332077133"
Oct 30 04:57:11.763816 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:57:11.763781    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:57:12.518640 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:12.518603    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz" event={"ID":"d9d290bd-4133-45ce-9816-a7b94e8d89a4","Type":"ContainerStarted","Data":"cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8"}
Oct 30 04:57:12.519064 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:12.518670    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz" event={"ID":"d9d290bd-4133-45ce-9816-a7b94e8d89a4","Type":"ContainerStarted","Data":"7ff76620fb44fa946a18dc1881750d0857b667ee694f93ded61b6afd99e04007"}
Oct 30 04:57:12.579795 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:12.579743    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz" podStartSLOduration=2.57972736 podStartE2EDuration="2.57972736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:12.578044556 +0000 UTC m=+3612.319467657" watchObservedRunningTime="2026-10-30 04:57:12.57972736 +0000 UTC m=+3612.321150458"
Oct 30 04:57:13.203691 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:13.203647    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:13.205329 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:13.205300    5072 patch_prober.go:29] interesting pod/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 04:57:13.205485 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:13.205358    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz" podUID="d9d290bd-4133-45ce-9816-a7b94e8d89a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 04:57:14.204319 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:14.204279    5072 patch_prober.go:29] interesting pod/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 04:57:14.204752 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:14.204335    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz" podUID="d9d290bd-4133-45ce-9816-a7b94e8d89a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 04:57:15.204917 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:15.204879    5072 patch_prober.go:29] interesting pod/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 04:57:15.205336 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:15.204935    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz" podUID="d9d290bd-4133-45ce-9816-a7b94e8d89a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 04:57:16.204596 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.204560    5072 patch_prober.go:29] interesting pod/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 04:57:16.204829 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.204661    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz" podUID="d9d290bd-4133-45ce-9816-a7b94e8d89a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 04:57:16.646598 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.646517    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"]
Oct 30 04:57:16.653216 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.653193    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"
Oct 30 04:57:16.686829 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.685444    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fd27a9-dockercfg-nnv5h\""
Oct 30 04:57:16.686829 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.686067    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fd27a9\""
Oct 30 04:57:16.694143 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.691923    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"]
Oct 30 04:57:16.811333 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.811299    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"]
Oct 30 04:57:16.847593 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.830517    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"
Oct 30 04:57:16.894189 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.893784    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk69w\" (UniqueName: \"kubernetes.io/projected/49ab9ae3-71d5-469d-8860-50240e90aa21-kube-api-access-fk69w\") pod \"payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq\" (UID: \"49ab9ae3-71d5-469d-8860-50240e90aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"
Oct 30 04:57:16.894189 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.893872    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d6aa4c-661b-4bb8-a051-87ed6f430464-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj\" (UID: \"71d6aa4c-661b-4bb8-a051-87ed6f430464\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"
Oct 30 04:57:16.911791 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.911312    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56br\" (UniqueName: \"kubernetes.io/projected/71d6aa4c-661b-4bb8-a051-87ed6f430464-kube-api-access-n56br\") pod \"payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj\" (UID: \"71d6aa4c-661b-4bb8-a051-87ed6f430464\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"
Oct 30 04:57:16.911791 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.911462    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ab9ae3-71d5-469d-8860-50240e90aa21-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq\" (UID: \"49ab9ae3-71d5-469d-8860-50240e90aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"
Oct 30 04:57:16.922165 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:16.918634    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"]
Oct 30 04:57:17.012701 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.012662    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56br\" (UniqueName: \"kubernetes.io/projected/71d6aa4c-661b-4bb8-a051-87ed6f430464-kube-api-access-n56br\") pod \"payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj\" (UID: \"71d6aa4c-661b-4bb8-a051-87ed6f430464\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"
Oct 30 04:57:17.012883 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.012797    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ab9ae3-71d5-469d-8860-50240e90aa21-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq\" (UID: \"49ab9ae3-71d5-469d-8860-50240e90aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"
Oct 30 04:57:17.012995 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.012973    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk69w\" (UniqueName: \"kubernetes.io/projected/49ab9ae3-71d5-469d-8860-50240e90aa21-kube-api-access-fk69w\") pod \"payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq\" (UID: \"49ab9ae3-71d5-469d-8860-50240e90aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"
Oct 30 04:57:17.013091 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.013058    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d6aa4c-661b-4bb8-a051-87ed6f430464-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj\" (UID: \"71d6aa4c-661b-4bb8-a051-87ed6f430464\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"
Oct 30 04:57:17.013660 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.013634    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ab9ae3-71d5-469d-8860-50240e90aa21-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq\" (UID: \"49ab9ae3-71d5-469d-8860-50240e90aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"
Oct 30 04:57:17.013814 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.013794    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d6aa4c-661b-4bb8-a051-87ed6f430464-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj\" (UID: \"71d6aa4c-661b-4bb8-a051-87ed6f430464\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"
Oct 30 04:57:17.040466 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.040431    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56br\" (UniqueName: \"kubernetes.io/projected/71d6aa4c-661b-4bb8-a051-87ed6f430464-kube-api-access-n56br\") pod \"payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj\" (UID: \"71d6aa4c-661b-4bb8-a051-87ed6f430464\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"
Oct 30 04:57:17.040609 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.040520    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk69w\" (UniqueName: \"kubernetes.io/projected/49ab9ae3-71d5-469d-8860-50240e90aa21-kube-api-access-fk69w\") pod \"payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq\" (UID: \"49ab9ae3-71d5-469d-8860-50240e90aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"
Oct 30 04:57:17.204030 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.203995    5072 patch_prober.go:29] interesting pod/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 04:57:17.204230 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.204048    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz" podUID="d9d290bd-4133-45ce-9816-a7b94e8d89a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 04:57:17.234854 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.234827    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"
Oct 30 04:57:17.309938 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.309838    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"
Oct 30 04:57:17.593116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.592731    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"]
Oct 30 04:57:17.606905 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.606015    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj" event={"ID":"71d6aa4c-661b-4bb8-a051-87ed6f430464","Type":"ContainerStarted","Data":"b6aaa302535e393b90337ec3dbe82fec58b6510f8a43910d2604eb695b6f51cf"}
Oct 30 04:57:17.687719 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:17.687690    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"]
Oct 30 04:57:17.690573 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:57:17.690539    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ab9ae3_71d5_469d_8860_50240e90aa21.slice/crio-7f0dbabdfed023900a460f28280e805a0aac2a624b20925bb7cf9890df392ba3 WatchSource:0}: Error finding container 7f0dbabdfed023900a460f28280e805a0aac2a624b20925bb7cf9890df392ba3: Status 404 returned error can't find the container with id 7f0dbabdfed023900a460f28280e805a0aac2a624b20925bb7cf9890df392ba3
Oct 30 04:57:18.204390 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:18.204355    5072 patch_prober.go:29] interesting pod/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 04:57:18.204574 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:18.204413    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz" podUID="d9d290bd-4133-45ce-9816-a7b94e8d89a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 04:57:18.619712 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:18.619608    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq" event={"ID":"49ab9ae3-71d5-469d-8860-50240e90aa21","Type":"ContainerStarted","Data":"3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e"}
Oct 30 04:57:18.619712 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:18.619703    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"
Oct 30 04:57:18.619948 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:18.619743    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq" event={"ID":"49ab9ae3-71d5-469d-8860-50240e90aa21","Type":"ContainerStarted","Data":"7f0dbabdfed023900a460f28280e805a0aac2a624b20925bb7cf9890df392ba3"}
Oct 30 04:57:18.621722 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:18.621688    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj" event={"ID":"71d6aa4c-661b-4bb8-a051-87ed6f430464","Type":"ContainerStarted","Data":"4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c"}
Oct 30 04:57:18.621860 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:18.621747    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"
Oct 30 04:57:18.715510 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:18.715446    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj" podStartSLOduration=2.715426686 podStartE2EDuration="2.715426686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:18.709694886 +0000 UTC m=+3618.451117984" watchObservedRunningTime="2026-10-30 04:57:18.715426686 +0000 UTC m=+3618.456849784"
Oct 30 04:57:18.716030 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:18.715919    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq" podStartSLOduration=2.715908282 podStartE2EDuration="2.715908282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:18.664251887 +0000 UTC m=+3618.405674985" watchObservedRunningTime="2026-10-30 04:57:18.715908282 +0000 UTC m=+3618.457331379"
Oct 30 04:57:19.210434 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:19.210397    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:19.210934 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:19.210921    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:19.211660 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:19.211636    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 04:57:22.522172 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:22.522142    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"
Oct 30 04:57:22.522575 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:22.522213    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"
Oct 30 04:57:29.633947 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:29.633905    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"
Oct 30 04:57:29.635184 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:29.635159    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"
Oct 30 04:57:31.033198 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:31.033162    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"]
Oct 30 04:57:31.045607 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:31.045574    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08d2931-s8kmt"]
Oct 30 04:57:32.415554 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:57:32.415515    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a330b420-dede-4577-865f-8d8a7fa944ad" path="/var/lib/kubelet/pods/a330b420-dede-4577-865f-8d8a7fa944ad/volumes"
Oct 30 04:57:41.753781 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:57:41.753660    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:58:02.009290 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:02.008619    5072 generic.go:356] "Generic (PLEG): container finished" podID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerID="764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501" exitCode=0
Oct 30 04:58:02.009290 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:02.008838    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" event={"ID":"1a006882-7f2d-4624-9bab-4d3623a0b75e","Type":"ContainerDied","Data":"764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501"}
Oct 30 04:58:02.009909 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:02.009395    5072 scope.go:122] "RemoveContainer" containerID="764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501"
Oct 30 04:58:02.034913 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:02.034862    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:58:02.035173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:02.034976    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:58:02.035173 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:02.035029    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:58:02.243262 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:02.243228    5072 scope.go:122] "RemoveContainer" containerID="9ae6a33c453849700518984e9b96b34e4abaeedc3abcc17699b5cbbd77d6e97d"
Oct 30 04:58:02.257999 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:02.257978    5072 scope.go:122] "RemoveContainer" containerID="2970de22336dd61257653a01a55994d499a4ffa022e915d96ca536507ca58069"
Oct 30 04:58:02.268683 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:02.268661    5072 scope.go:122] "RemoveContainer" containerID="caab2887e92f2678aa9db511df7277282946716a7a7b3e02f4ab88430655c44c"
Oct 30 04:58:02.280241 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:02.280217    5072 scope.go:122] "RemoveContainer" containerID="7c454bfdfb60f1aad85a4baa6ed09e890a17d5c7fb6e95d6c934832021b41510"
Oct 30 04:58:02.290273 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:02.290249    5072 scope.go:122] "RemoveContainer" containerID="53d96b3b9476e7b21dbcbb7e5ac13b7b9c420c782a64d5240e74f8e9bb876452"
Oct 30 04:58:02.300188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:02.300169    5072 scope.go:122] "RemoveContainer" containerID="334872e6a23d73d1557617bde9973355c0ff3ce579d42868fa0f2d6193f37082"
Oct 30 04:58:03.018961 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:03.018915    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" event={"ID":"1a006882-7f2d-4624-9bab-4d3623a0b75e","Type":"ContainerStarted","Data":"3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a"}
Oct 30 04:58:03.634179 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:03.634144    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:58:03.634389 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:03.634360    5072 patch_prober.go:29] interesting pod/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 04:58:03.634467 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:03.634434    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 04:58:04.634401 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:04.634366    5072 patch_prober.go:29] interesting pod/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 04:58:04.634829 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:04.634426    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 04:58:05.634225 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:05.634187    5072 patch_prober.go:29] interesting pod/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 04:58:05.634394 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:05.634254    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 04:58:06.634043 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:06.634011    5072 patch_prober.go:29] interesting pod/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 04:58:06.634461 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:06.634085    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 04:58:07.634046 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:07.634009    5072 patch_prober.go:29] interesting pod/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 04:58:07.634536 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:07.634094    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 04:58:08.663812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:08.662819    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:58:08.663812 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:08.663409    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:58:08.664622 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:08.664602    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 04:58:11.753474 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:58:11.753436    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:58:26.076368 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.076330    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"]
Oct 30 04:58:26.082287 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.082261    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"
Oct 30 04:58:26.098505 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.098477    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e592d7\""
Oct 30 04:58:26.098702 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.098501    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e592d7-dockercfg-sqd5r\""
Oct 30 04:58:26.116671 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.116639    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"]
Oct 30 04:58:26.241551 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.241521    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"]
Oct 30 04:58:26.246923 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.246885    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swgpm\" (UniqueName: \"kubernetes.io/projected/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-kube-api-access-swgpm\") pod \"payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v\" (UID: \"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"
Oct 30 04:58:26.247191 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.247169    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v\" (UID: \"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"
Oct 30 04:58:26.250725 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.250705    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"
Oct 30 04:58:26.255843 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.251694    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"]
Oct 30 04:58:26.349222 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.349144    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7pk\" (UniqueName: \"kubernetes.io/projected/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-kube-api-access-sj7pk\") pod \"payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh\" (UID: \"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"
Oct 30 04:58:26.349222 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.349200    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v\" (UID: \"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"
Oct 30 04:58:26.349468 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.349396    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh\" (UID: \"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"
Oct 30 04:58:26.349516 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.349491    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swgpm\" (UniqueName: \"kubernetes.io/projected/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-kube-api-access-swgpm\") pod \"payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v\" (UID: \"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"
Oct 30 04:58:26.350214 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.350191    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v\" (UID: \"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"
Oct 30 04:58:26.400492 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.400457    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swgpm\" (UniqueName: \"kubernetes.io/projected/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-kube-api-access-swgpm\") pod \"payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v\" (UID: \"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"
Oct 30 04:58:26.402748 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.402720    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"
Oct 30 04:58:26.451094 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.451023    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7pk\" (UniqueName: \"kubernetes.io/projected/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-kube-api-access-sj7pk\") pod \"payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh\" (UID: \"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"
Oct 30 04:58:26.451507 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.451483    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh\" (UID: \"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"
Oct 30 04:58:26.452240 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.452219    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh\" (UID: \"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"
Oct 30 04:58:26.477476 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.477297    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7pk\" (UniqueName: \"kubernetes.io/projected/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-kube-api-access-sj7pk\") pod \"payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh\" (UID: \"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"
Oct 30 04:58:26.589484 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.589449    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"
Oct 30 04:58:26.715099 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:58:26.715043    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a61dd8_9bd0_4ec2_bffd_0bfb4d03a282.slice/crio-c9cb6a2141de9117e506a4231da0bf406669e7e6006c552bbe7d58dcdfdf9c95 WatchSource:0}: Error finding container c9cb6a2141de9117e506a4231da0bf406669e7e6006c552bbe7d58dcdfdf9c95: Status 404 returned error can't find the container with id c9cb6a2141de9117e506a4231da0bf406669e7e6006c552bbe7d58dcdfdf9c95
Oct 30 04:58:26.715345 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.715316    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"]
Oct 30 04:58:26.902446 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:58:26.902401    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d1b5c8_fa40_4c9b_8a93_c29c13595d3a.slice/crio-295ba4f21afaa137101459e692142f54591661bf79c7e23bf6b789a95beb68c7 WatchSource:0}: Error finding container 295ba4f21afaa137101459e692142f54591661bf79c7e23bf6b789a95beb68c7: Status 404 returned error can't find the container with id 295ba4f21afaa137101459e692142f54591661bf79c7e23bf6b789a95beb68c7
Oct 30 04:58:26.903591 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:26.903565    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"]
Oct 30 04:58:27.298241 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:27.298198    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v" event={"ID":"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282","Type":"ContainerStarted","Data":"16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b"}
Oct 30 04:58:27.298710 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:27.298267    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"
Oct 30 04:58:27.298710 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:27.298289    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v" event={"ID":"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282","Type":"ContainerStarted","Data":"c9cb6a2141de9117e506a4231da0bf406669e7e6006c552bbe7d58dcdfdf9c95"}
Oct 30 04:58:27.300022 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:27.299995    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh" event={"ID":"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a","Type":"ContainerStarted","Data":"f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14"}
Oct 30 04:58:27.300188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:27.300047    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"
Oct 30 04:58:27.300188 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:27.300064    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh" event={"ID":"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a","Type":"ContainerStarted","Data":"295ba4f21afaa137101459e692142f54591661bf79c7e23bf6b789a95beb68c7"}
Oct 30 04:58:27.344675 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:27.344630    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v" podStartSLOduration=1.3446169239999999 podStartE2EDuration="1.344616924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:27.340300157 +0000 UTC m=+3687.081723254" watchObservedRunningTime="2026-10-30 04:58:27.344616924 +0000 UTC m=+3687.086040021"
Oct 30 04:58:27.384116 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:27.384040    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh" podStartSLOduration=1.384023252 podStartE2EDuration="1.384023252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:27.382176104 +0000 UTC m=+3687.123599214" watchObservedRunningTime="2026-10-30 04:58:27.384023252 +0000 UTC m=+3687.125446350"
Oct 30 04:58:38.312596 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:38.312565    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"
Oct 30 04:58:38.313098 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:58:38.312632    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"
Oct 30 04:58:41.753975 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:58:41.753938    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:59:02.389666 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:02.389628    5072 scope.go:122] "RemoveContainer" containerID="9b8efbbfd797559d0b1cda46c60e5124f4989b5782490666eeae08f2f69b1165"
Oct 30 04:59:07.726045 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:07.726003    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"]
Oct 30 04:59:07.732787 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:07.732747    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"
Oct 30 04:59:07.746654 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:07.746562    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"]
Oct 30 04:59:07.753511 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:07.753460    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-26b72f\""
Oct 30 04:59:07.753991 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:07.753973    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-26b72f-dockercfg-st896\""
Oct 30 04:59:07.789479 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:07.789446    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934a686e-a82d-403e-b13f-97637b888cee-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm\" (UID: \"934a686e-a82d-403e-b13f-97637b888cee\") " pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"
Oct 30 04:59:07.789955 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:07.789927    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pblfh\" (UniqueName: \"kubernetes.io/projected/934a686e-a82d-403e-b13f-97637b888cee-kube-api-access-pblfh\") pod \"payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm\" (UID: \"934a686e-a82d-403e-b13f-97637b888cee\") " pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"
Oct 30 04:59:07.891141 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:07.891099    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934a686e-a82d-403e-b13f-97637b888cee-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm\" (UID: \"934a686e-a82d-403e-b13f-97637b888cee\") " pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"
Oct 30 04:59:07.891357 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:07.891317    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pblfh\" (UniqueName: \"kubernetes.io/projected/934a686e-a82d-403e-b13f-97637b888cee-kube-api-access-pblfh\") pod \"payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm\" (UID: \"934a686e-a82d-403e-b13f-97637b888cee\") " pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"
Oct 30 04:59:07.891931 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:07.891902    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934a686e-a82d-403e-b13f-97637b888cee-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm\" (UID: \"934a686e-a82d-403e-b13f-97637b888cee\") " pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"
Oct 30 04:59:07.929932 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:07.921393    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pblfh\" (UniqueName: \"kubernetes.io/projected/934a686e-a82d-403e-b13f-97637b888cee-kube-api-access-pblfh\") pod \"payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm\" (UID: \"934a686e-a82d-403e-b13f-97637b888cee\") " pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"
Oct 30 04:59:08.052152 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:08.051691    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"
Oct 30 04:59:08.429533 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:08.429497    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"]
Oct 30 04:59:08.436362 ip-10-0-139-99 kubenswrapper[5072]: W1030 04:59:08.436333    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934a686e_a82d_403e_b13f_97637b888cee.slice/crio-dc7b8c124b92f860e8cada5a0b9860b75c85979d8cf0f5c62ecf2c843c9e47a7 WatchSource:0}: Error finding container dc7b8c124b92f860e8cada5a0b9860b75c85979d8cf0f5c62ecf2c843c9e47a7: Status 404 returned error can't find the container with id dc7b8c124b92f860e8cada5a0b9860b75c85979d8cf0f5c62ecf2c843c9e47a7
Oct 30 04:59:08.714250 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:08.714198    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm" event={"ID":"934a686e-a82d-403e-b13f-97637b888cee","Type":"ContainerStarted","Data":"49c727f733b38bb310472c23b22e402c0668f114e443b99407bf893eddbb7cf6"}
Oct 30 04:59:08.714463 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:08.714279    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm" event={"ID":"934a686e-a82d-403e-b13f-97637b888cee","Type":"ContainerStarted","Data":"dc7b8c124b92f860e8cada5a0b9860b75c85979d8cf0f5c62ecf2c843c9e47a7"}
Oct 30 04:59:08.714463 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:08.714322    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"
Oct 30 04:59:08.778425 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:08.778374    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm" podStartSLOduration=1.7783589869999998 podStartE2EDuration="1.778358987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:08.774770346 +0000 UTC m=+3728.516193444" watchObservedRunningTime="2026-10-30 04:59:08.778358987 +0000 UTC m=+3728.519782083"
Oct 30 04:59:11.754475 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:59:11.754395    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:59:19.727481 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:19.727453    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"
Oct 30 04:59:41.753384 ip-10-0-139-99 kubenswrapper[5072]: E1030 04:59:41.753338    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 04:59:55.032680 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:55.032640    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"]
Oct 30 04:59:55.044924 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:55.044891    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4403c82-wmlz7"]
Oct 30 04:59:56.411325 ip-10-0-139-99 kubenswrapper[5072]: I1030 04:59:56.411295    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9100522a-a086-4a10-8ae8-962c23ab0cc4" path="/var/lib/kubelet/pods/9100522a-a086-4a10-8ae8-962c23ab0cc4/volumes"
Oct 30 05:00:00.150507 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.150469    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"]
Oct 30 05:00:00.158325 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.158298    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"]
Oct 30 05:00:00.158488 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.158402    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"
Oct 30 05:00:00.163273 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.163244    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"]
Oct 30 05:00:00.163420 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.163373    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"
Oct 30 05:00:00.167998 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.167978    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"]
Oct 30 05:00:00.168116 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.168069    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"
Oct 30 05:00:00.174028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.174004    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"]
Oct 30 05:00:00.175895 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.175873    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"
Oct 30 05:00:00.177699 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.177648    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:00:00.187854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.181460    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"
Oct 30 05:00:00.221006 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.220970    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"]
Oct 30 05:00:00.236103 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.233154    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"]
Oct 30 05:00:00.244253 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.244220    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"]
Oct 30 05:00:00.247473 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.246773    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"]
Oct 30 05:00:00.252321 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.252290    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9\" (UID: \"6fa98861-f2c2-47aa-bb83-54ddf8c5c715\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"
Oct 30 05:00:00.252477 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.252336    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqz4x\" (UniqueName: \"kubernetes.io/projected/f52b50b1-dfe7-4316-9918-61d44366cd60-kube-api-access-mqz4x\") pod \"maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv\" (UID: \"f52b50b1-dfe7-4316-9918-61d44366cd60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"
Oct 30 05:00:00.252546 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.252491    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv874\" (UniqueName: \"kubernetes.io/projected/a5cb00e5-a10c-4e5a-846b-7e10e452611e-kube-api-access-qv874\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d\" (UID: \"a5cb00e5-a10c-4e5a-846b-7e10e452611e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"
Oct 30 05:00:00.252660 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.252577    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vbgb\" (UniqueName: \"kubernetes.io/projected/3621a9d6-ea49-415f-a6a9-5ce99245d111-kube-api-access-4vbgb\") pod \"maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n\" (UID: \"3621a9d6-ea49-415f-a6a9-5ce99245d111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"
Oct 30 05:00:00.252660 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.252634    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48d64\" (UniqueName: \"kubernetes.io/projected/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-kube-api-access-48d64\") pod \"maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9\" (UID: \"6fa98861-f2c2-47aa-bb83-54ddf8c5c715\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"
Oct 30 05:00:00.252774 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.252669    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5cb00e5-a10c-4e5a-846b-7e10e452611e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d\" (UID: \"a5cb00e5-a10c-4e5a-846b-7e10e452611e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"
Oct 30 05:00:00.252774 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.252699    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e82ab9-06b8-4c71-b46a-76348dee31b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb\" (UID: \"92e82ab9-06b8-4c71-b46a-76348dee31b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"
Oct 30 05:00:00.252885 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.252852    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3621a9d6-ea49-415f-a6a9-5ce99245d111-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n\" (UID: \"3621a9d6-ea49-415f-a6a9-5ce99245d111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"
Oct 30 05:00:00.252942 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.252918    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52b50b1-dfe7-4316-9918-61d44366cd60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv\" (UID: \"f52b50b1-dfe7-4316-9918-61d44366cd60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"
Oct 30 05:00:00.253003 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.252949    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdgsv\" (UniqueName: \"kubernetes.io/projected/92e82ab9-06b8-4c71-b46a-76348dee31b5-kube-api-access-rdgsv\") pod \"maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb\" (UID: \"92e82ab9-06b8-4c71-b46a-76348dee31b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"
Oct 30 05:00:00.265822 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.265794    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"
Oct 30 05:00:00.265958 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.265698    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"]
Oct 30 05:00:00.266026 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.265995    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"]
Oct 30 05:00:00.279617 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.279573    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"]
Oct 30 05:00:00.279835 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.279663    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"
Oct 30 05:00:00.295366 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.295344    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"]
Oct 30 05:00:00.296571 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.296545    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"
Oct 30 05:00:00.311295 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.311029    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"]
Oct 30 05:00:00.312225 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.311143    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"
Oct 30 05:00:00.320060 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.320018    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"]
Oct 30 05:00:00.320060 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.320057    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"]
Oct 30 05:00:00.320461 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.320127    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"
Oct 30 05:00:00.334349 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.332795    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"]
Oct 30 05:00:00.334349 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.332828    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"]
Oct 30 05:00:00.334349 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.332899    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"
Oct 30 05:00:00.341473 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.341411    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"]
Oct 30 05:00:00.351811 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.351790    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"]
Oct 30 05:00:00.354419 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.354390    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48d64\" (UniqueName: \"kubernetes.io/projected/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-kube-api-access-48d64\") pod \"maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9\" (UID: \"6fa98861-f2c2-47aa-bb83-54ddf8c5c715\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"
Oct 30 05:00:00.355543 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.355519    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5cb00e5-a10c-4e5a-846b-7e10e452611e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d\" (UID: \"a5cb00e5-a10c-4e5a-846b-7e10e452611e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"
Oct 30 05:00:00.355708 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.355690    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e82ab9-06b8-4c71-b46a-76348dee31b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb\" (UID: \"92e82ab9-06b8-4c71-b46a-76348dee31b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"
Oct 30 05:00:00.355921 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.355906    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3621a9d6-ea49-415f-a6a9-5ce99245d111-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n\" (UID: \"3621a9d6-ea49-415f-a6a9-5ce99245d111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"
Oct 30 05:00:00.356040 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.356025    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212e51ce-0a69-4809-822f-6f6fe6bf8773-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9\" (UID: \"212e51ce-0a69-4809-822f-6f6fe6bf8773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"
Oct 30 05:00:00.356186 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.356171    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/804abcc6-7570-4f94-be22-d2c8b12f2108-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h\" (UID: \"804abcc6-7570-4f94-be22-d2c8b12f2108\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"
Oct 30 05:00:00.356325 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.356311    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52b50b1-dfe7-4316-9918-61d44366cd60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv\" (UID: \"f52b50b1-dfe7-4316-9918-61d44366cd60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"
Oct 30 05:00:00.356445 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.356419    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdgsv\" (UniqueName: \"kubernetes.io/projected/92e82ab9-06b8-4c71-b46a-76348dee31b5-kube-api-access-rdgsv\") pod \"maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb\" (UID: \"92e82ab9-06b8-4c71-b46a-76348dee31b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"
Oct 30 05:00:00.356576 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.356560    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9\" (UID: \"6fa98861-f2c2-47aa-bb83-54ddf8c5c715\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"
Oct 30 05:00:00.356679 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.356666    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqz4x\" (UniqueName: \"kubernetes.io/projected/f52b50b1-dfe7-4316-9918-61d44366cd60-kube-api-access-mqz4x\") pod \"maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv\" (UID: \"f52b50b1-dfe7-4316-9918-61d44366cd60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"
Oct 30 05:00:00.356788 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.356773    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4pqf\" (UniqueName: \"kubernetes.io/projected/212e51ce-0a69-4809-822f-6f6fe6bf8773-kube-api-access-s4pqf\") pod \"maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9\" (UID: \"212e51ce-0a69-4809-822f-6f6fe6bf8773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"
Oct 30 05:00:00.356908 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.356893    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqn8x\" (UniqueName: \"kubernetes.io/projected/804abcc6-7570-4f94-be22-d2c8b12f2108-kube-api-access-xqn8x\") pod \"maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h\" (UID: \"804abcc6-7570-4f94-be22-d2c8b12f2108\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"
Oct 30 05:00:00.357025 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.357012    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/927eaf22-dd8b-4bef-a48c-fff361ead463-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m\" (UID: \"927eaf22-dd8b-4bef-a48c-fff361ead463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"
Oct 30 05:00:00.357189 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.357172    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53d6987-db5e-468a-9afe-2c0603cbf408-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5\" (UID: \"e53d6987-db5e-468a-9afe-2c0603cbf408\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"
Oct 30 05:00:00.357366 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.357351    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv874\" (UniqueName: \"kubernetes.io/projected/a5cb00e5-a10c-4e5a-846b-7e10e452611e-kube-api-access-qv874\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d\" (UID: \"a5cb00e5-a10c-4e5a-846b-7e10e452611e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"
Oct 30 05:00:00.357495 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.357464    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb9vw\" (UniqueName: \"kubernetes.io/projected/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-kube-api-access-zb9vw\") pod \"maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk\" (UID: \"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"
Oct 30 05:00:00.357604 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.357587    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qksl6\" (UniqueName: \"kubernetes.io/projected/927eaf22-dd8b-4bef-a48c-fff361ead463-kube-api-access-qksl6\") pod \"maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m\" (UID: \"927eaf22-dd8b-4bef-a48c-fff361ead463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"
Oct 30 05:00:00.357766 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.357751    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk\" (UID: \"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"
Oct 30 05:00:00.357875 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.357862    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vbgb\" (UniqueName: \"kubernetes.io/projected/3621a9d6-ea49-415f-a6a9-5ce99245d111-kube-api-access-4vbgb\") pod \"maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n\" (UID: \"3621a9d6-ea49-415f-a6a9-5ce99245d111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"
Oct 30 05:00:00.357993 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.357980    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqx7p\" (UniqueName: \"kubernetes.io/projected/e53d6987-db5e-468a-9afe-2c0603cbf408-kube-api-access-sqx7p\") pod \"maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5\" (UID: \"e53d6987-db5e-468a-9afe-2c0603cbf408\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"
Oct 30 05:00:00.359134 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.359113    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5cb00e5-a10c-4e5a-846b-7e10e452611e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d\" (UID: \"a5cb00e5-a10c-4e5a-846b-7e10e452611e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"
Oct 30 05:00:00.359275 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.359208    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"]
Oct 30 05:00:00.360010 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.359991    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e82ab9-06b8-4c71-b46a-76348dee31b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb\" (UID: \"92e82ab9-06b8-4c71-b46a-76348dee31b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"
Oct 30 05:00:00.360795 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.360773    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3621a9d6-ea49-415f-a6a9-5ce99245d111-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n\" (UID: \"3621a9d6-ea49-415f-a6a9-5ce99245d111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"
Oct 30 05:00:00.362131 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.361629    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52b50b1-dfe7-4316-9918-61d44366cd60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv\" (UID: \"f52b50b1-dfe7-4316-9918-61d44366cd60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"
Oct 30 05:00:00.362510 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.362482    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9\" (UID: \"6fa98861-f2c2-47aa-bb83-54ddf8c5c715\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"
Oct 30 05:00:00.441346 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.439932    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqz4x\" (UniqueName: \"kubernetes.io/projected/f52b50b1-dfe7-4316-9918-61d44366cd60-kube-api-access-mqz4x\") pod \"maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv\" (UID: \"f52b50b1-dfe7-4316-9918-61d44366cd60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"
Oct 30 05:00:00.441346 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.440806    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48d64\" (UniqueName: \"kubernetes.io/projected/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-kube-api-access-48d64\") pod \"maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9\" (UID: \"6fa98861-f2c2-47aa-bb83-54ddf8c5c715\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"
Oct 30 05:00:00.441346 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.441056    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv874\" (UniqueName: \"kubernetes.io/projected/a5cb00e5-a10c-4e5a-846b-7e10e452611e-kube-api-access-qv874\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d\" (UID: \"a5cb00e5-a10c-4e5a-846b-7e10e452611e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"
Oct 30 05:00:00.441627 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.441429    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdgsv\" (UniqueName: \"kubernetes.io/projected/92e82ab9-06b8-4c71-b46a-76348dee31b5-kube-api-access-rdgsv\") pod \"maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb\" (UID: \"92e82ab9-06b8-4c71-b46a-76348dee31b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"
Oct 30 05:00:00.445777 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.445745    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"]
Oct 30 05:00:00.452010 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.451984    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vbgb\" (UniqueName: \"kubernetes.io/projected/3621a9d6-ea49-415f-a6a9-5ce99245d111-kube-api-access-4vbgb\") pod \"maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n\" (UID: \"3621a9d6-ea49-415f-a6a9-5ce99245d111\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.459350    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212e51ce-0a69-4809-822f-6f6fe6bf8773-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9\" (UID: \"212e51ce-0a69-4809-822f-6f6fe6bf8773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.459393    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/804abcc6-7570-4f94-be22-d2c8b12f2108-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h\" (UID: \"804abcc6-7570-4f94-be22-d2c8b12f2108\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.459475    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4pqf\" (UniqueName: \"kubernetes.io/projected/212e51ce-0a69-4809-822f-6f6fe6bf8773-kube-api-access-s4pqf\") pod \"maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9\" (UID: \"212e51ce-0a69-4809-822f-6f6fe6bf8773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.459514    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqn8x\" (UniqueName: \"kubernetes.io/projected/804abcc6-7570-4f94-be22-d2c8b12f2108-kube-api-access-xqn8x\") pod \"maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h\" (UID: \"804abcc6-7570-4f94-be22-d2c8b12f2108\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.459557    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/927eaf22-dd8b-4bef-a48c-fff361ead463-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m\" (UID: \"927eaf22-dd8b-4bef-a48c-fff361ead463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.459597    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8802708-e588-476f-b55b-bce1a3efa0cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw\" (UID: \"d8802708-e588-476f-b55b-bce1a3efa0cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.459649    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53d6987-db5e-468a-9afe-2c0603cbf408-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5\" (UID: \"e53d6987-db5e-468a-9afe-2c0603cbf408\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.459728    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb9vw\" (UniqueName: \"kubernetes.io/projected/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-kube-api-access-zb9vw\") pod \"maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk\" (UID: \"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.459754    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qksl6\" (UniqueName: \"kubernetes.io/projected/927eaf22-dd8b-4bef-a48c-fff361ead463-kube-api-access-qksl6\") pod \"maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m\" (UID: \"927eaf22-dd8b-4bef-a48c-fff361ead463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.459832    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk\" (UID: \"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.459880    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx7p\" (UniqueName: \"kubernetes.io/projected/e53d6987-db5e-468a-9afe-2c0603cbf408-kube-api-access-sqx7p\") pod \"maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5\" (UID: \"e53d6987-db5e-468a-9afe-2c0603cbf408\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.459975    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m257r\" (UniqueName: \"kubernetes.io/projected/d8802708-e588-476f-b55b-bce1a3efa0cd-kube-api-access-m257r\") pod \"maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw\" (UID: \"d8802708-e588-476f-b55b-bce1a3efa0cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"
Oct 30 05:00:00.461179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.461031    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/927eaf22-dd8b-4bef-a48c-fff361ead463-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m\" (UID: \"927eaf22-dd8b-4bef-a48c-fff361ead463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"
Oct 30 05:00:00.461955 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.461431    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212e51ce-0a69-4809-822f-6f6fe6bf8773-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9\" (UID: \"212e51ce-0a69-4809-822f-6f6fe6bf8773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"
Oct 30 05:00:00.462150 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.462106    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/804abcc6-7570-4f94-be22-d2c8b12f2108-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h\" (UID: \"804abcc6-7570-4f94-be22-d2c8b12f2108\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"
Oct 30 05:00:00.463462 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.462805    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk\" (UID: \"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"
Oct 30 05:00:00.463462 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.462836    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53d6987-db5e-468a-9afe-2c0603cbf408-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5\" (UID: \"e53d6987-db5e-468a-9afe-2c0603cbf408\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"
Oct 30 05:00:00.469607 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.469581    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"]
Oct 30 05:00:00.469731 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.469709    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"
Oct 30 05:00:00.474543 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.474525    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"]
Oct 30 05:00:00.474661 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.474572    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"]
Oct 30 05:00:00.474875 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.474858    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"
Oct 30 05:00:00.479667 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.479650    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"]
Oct 30 05:00:00.479779 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.479760    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"
Oct 30 05:00:00.485491 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.485468    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"]
Oct 30 05:00:00.485600 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.485510    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"]
Oct 30 05:00:00.485600 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.485537    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"]
Oct 30 05:00:00.485600 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.485562    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"]
Oct 30 05:00:00.486747 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.486722    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqn8x\" (UniqueName: \"kubernetes.io/projected/804abcc6-7570-4f94-be22-d2c8b12f2108-kube-api-access-xqn8x\") pod \"maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h\" (UID: \"804abcc6-7570-4f94-be22-d2c8b12f2108\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"
Oct 30 05:00:00.487327 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.487299    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4pqf\" (UniqueName: \"kubernetes.io/projected/212e51ce-0a69-4809-822f-6f6fe6bf8773-kube-api-access-s4pqf\") pod \"maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9\" (UID: \"212e51ce-0a69-4809-822f-6f6fe6bf8773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"
Oct 30 05:00:00.487486 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.487463    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"
Oct 30 05:00:00.488786 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.488761    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb9vw\" (UniqueName: \"kubernetes.io/projected/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-kube-api-access-zb9vw\") pod \"maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk\" (UID: \"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"
Oct 30 05:00:00.489982 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.489958    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqx7p\" (UniqueName: \"kubernetes.io/projected/e53d6987-db5e-468a-9afe-2c0603cbf408-kube-api-access-sqx7p\") pod \"maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5\" (UID: \"e53d6987-db5e-468a-9afe-2c0603cbf408\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"
Oct 30 05:00:00.492012 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.491956    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"
Oct 30 05:00:00.492230 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.492211    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"]
Oct 30 05:00:00.493880 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.493862    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qksl6\" (UniqueName: \"kubernetes.io/projected/927eaf22-dd8b-4bef-a48c-fff361ead463-kube-api-access-qksl6\") pod \"maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m\" (UID: \"927eaf22-dd8b-4bef-a48c-fff361ead463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"
Oct 30 05:00:00.495493 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.495476    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:00:00.498601 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.498579    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"]
Oct 30 05:00:00.498700 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.498638    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"
Oct 30 05:00:00.500968 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.500947    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"
Oct 30 05:00:00.507776 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.504620    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"]
Oct 30 05:00:00.507776 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.504649    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"]
Oct 30 05:00:00.507776 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.504702    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"
Oct 30 05:00:00.510415 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.510381    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"]
Oct 30 05:00:00.518454 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.518435    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"]
Oct 30 05:00:00.518623 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.518607    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"
Oct 30 05:00:00.524518 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.524494    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"]
Oct 30 05:00:00.524667 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.524583    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"
Oct 30 05:00:00.532001 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.530704    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"]
Oct 30 05:00:00.537184 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.537126    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"]
Oct 30 05:00:00.542468 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.542446    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"]
Oct 30 05:00:00.547261 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.546434    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"]
Oct 30 05:00:00.550351 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.550324    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"
Oct 30 05:00:00.553553 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.553476    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"
Oct 30 05:00:00.558266 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.558245    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"]
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.561547    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ca5f62-df90-42b8-91cb-dadacd5b3668-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7\" (UID: \"82ca5f62-df90-42b8-91cb-dadacd5b3668\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.561588    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f638f03d-08bb-4937-aa89-2198ac4a4c02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g\" (UID: \"f638f03d-08bb-4937-aa89-2198ac4a4c02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.561676    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8802708-e588-476f-b55b-bce1a3efa0cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw\" (UID: \"d8802708-e588-476f-b55b-bce1a3efa0cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.561747    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt\" (UID: \"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.561777    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9acd468-3d02-4521-9db7-33e9dff0259f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs\" (UID: \"c9acd468-3d02-4521-9db7-33e9dff0259f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.561875    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8n6b\" (UniqueName: \"kubernetes.io/projected/82ca5f62-df90-42b8-91cb-dadacd5b3668-kube-api-access-j8n6b\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7\" (UID: \"82ca5f62-df90-42b8-91cb-dadacd5b3668\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.561924    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcvhb\" (UniqueName: \"kubernetes.io/projected/c9acd468-3d02-4521-9db7-33e9dff0259f-kube-api-access-vcvhb\") pod \"maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs\" (UID: \"c9acd468-3d02-4521-9db7-33e9dff0259f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.561963    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmn5c\" (UniqueName: \"kubernetes.io/projected/7a540e59-8239-423a-ac80-aa84d7b4821d-kube-api-access-qmn5c\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw\" (UID: \"7a540e59-8239-423a-ac80-aa84d7b4821d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.561989    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnvh6\" (UniqueName: \"kubernetes.io/projected/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-kube-api-access-pnvh6\") pod \"maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt\" (UID: \"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.562043    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e846f7c-186a-4e49-ba15-6ff6c564bc13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6\" (UID: \"0e846f7c-186a-4e49-ba15-6ff6c564bc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.562116    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a82ff41-ac42-48bc-81c4-24e539017282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf\" (UID: \"2a82ff41-ac42-48bc-81c4-24e539017282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.562163    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc366c3b-6526-4575-899f-df191a5971de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2\" (UID: \"fc366c3b-6526-4575-899f-df191a5971de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.562207    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m257r\" (UniqueName: \"kubernetes.io/projected/d8802708-e588-476f-b55b-bce1a3efa0cd-kube-api-access-m257r\") pod \"maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw\" (UID: \"d8802708-e588-476f-b55b-bce1a3efa0cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.562301    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmn4h\" (UniqueName: \"kubernetes.io/projected/5d6f868c-5bea-49e5-a136-ab5862d1841d-kube-api-access-pmn4h\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c\" (UID: \"5d6f868c-5bea-49e5-a136-ab5862d1841d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.562340    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5b4f\" (UniqueName: \"kubernetes.io/projected/f638f03d-08bb-4937-aa89-2198ac4a4c02-kube-api-access-t5b4f\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g\" (UID: \"f638f03d-08bb-4937-aa89-2198ac4a4c02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.562415    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csgjj\" (UniqueName: \"kubernetes.io/projected/fc366c3b-6526-4575-899f-df191a5971de-kube-api-access-csgjj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2\" (UID: \"fc366c3b-6526-4575-899f-df191a5971de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.562450    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6f868c-5bea-49e5-a136-ab5862d1841d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c\" (UID: \"5d6f868c-5bea-49e5-a136-ab5862d1841d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.562552    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kd87\" (UniqueName: \"kubernetes.io/projected/0e846f7c-186a-4e49-ba15-6ff6c564bc13-kube-api-access-4kd87\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6\" (UID: \"0e846f7c-186a-4e49-ba15-6ff6c564bc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.562578    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a540e59-8239-423a-ac80-aa84d7b4821d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw\" (UID: \"7a540e59-8239-423a-ac80-aa84d7b4821d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"
Oct 30 05:00:00.562861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.562615    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nlf\" (UniqueName: \"kubernetes.io/projected/2a82ff41-ac42-48bc-81c4-24e539017282-kube-api-access-84nlf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf\" (UID: \"2a82ff41-ac42-48bc-81c4-24e539017282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"
Oct 30 05:00:00.564026 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.563934    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8802708-e588-476f-b55b-bce1a3efa0cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw\" (UID: \"d8802708-e588-476f-b55b-bce1a3efa0cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"
Oct 30 05:00:00.569537 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.569503    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"]
Oct 30 05:00:00.569647 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.569612    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"
Oct 30 05:00:00.573021 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.573003    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"]
Oct 30 05:00:00.591053 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.590297    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m257r\" (UniqueName: \"kubernetes.io/projected/d8802708-e588-476f-b55b-bce1a3efa0cd-kube-api-access-m257r\") pod \"maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw\" (UID: \"d8802708-e588-476f-b55b-bce1a3efa0cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"
Oct 30 05:00:00.604649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.602455    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"
Oct 30 05:00:00.629091 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.612611    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"
Oct 30 05:00:00.642225 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.640524    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"
Oct 30 05:00:00.647805 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.646962    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"
Oct 30 05:00:00.664240 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664209    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc366c3b-6526-4575-899f-df191a5971de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2\" (UID: \"fc366c3b-6526-4575-899f-df191a5971de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"
Oct 30 05:00:00.664363 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664285    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmn4h\" (UniqueName: \"kubernetes.io/projected/5d6f868c-5bea-49e5-a136-ab5862d1841d-kube-api-access-pmn4h\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c\" (UID: \"5d6f868c-5bea-49e5-a136-ab5862d1841d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"
Oct 30 05:00:00.664363 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664307    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5b4f\" (UniqueName: \"kubernetes.io/projected/f638f03d-08bb-4937-aa89-2198ac4a4c02-kube-api-access-t5b4f\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g\" (UID: \"f638f03d-08bb-4937-aa89-2198ac4a4c02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"
Oct 30 05:00:00.664363 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664344    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgjj\" (UniqueName: \"kubernetes.io/projected/fc366c3b-6526-4575-899f-df191a5971de-kube-api-access-csgjj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2\" (UID: \"fc366c3b-6526-4575-899f-df191a5971de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"
Oct 30 05:00:00.664480 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664366    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6f868c-5bea-49e5-a136-ab5862d1841d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c\" (UID: \"5d6f868c-5bea-49e5-a136-ab5862d1841d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"
Oct 30 05:00:00.664480 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664393    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888940-bbnrh\" (UID: \"7fdb04a5-7116-41a3-80cd-f1169b75cc2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"
Oct 30 05:00:00.664480 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664423    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68kcr\" (UniqueName: \"kubernetes.io/projected/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-kube-api-access-68kcr\") pod \"maas-api-key-cleanup-29888940-bbnrh\" (UID: \"7fdb04a5-7116-41a3-80cd-f1169b75cc2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"
Oct 30 05:00:00.664480 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664452    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kd87\" (UniqueName: \"kubernetes.io/projected/0e846f7c-186a-4e49-ba15-6ff6c564bc13-kube-api-access-4kd87\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6\" (UID: \"0e846f7c-186a-4e49-ba15-6ff6c564bc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"
Oct 30 05:00:00.664480 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664467    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a540e59-8239-423a-ac80-aa84d7b4821d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw\" (UID: \"7a540e59-8239-423a-ac80-aa84d7b4821d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"
Oct 30 05:00:00.664649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664487    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nlf\" (UniqueName: \"kubernetes.io/projected/2a82ff41-ac42-48bc-81c4-24e539017282-kube-api-access-84nlf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf\" (UID: \"2a82ff41-ac42-48bc-81c4-24e539017282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"
Oct 30 05:00:00.664649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664519    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ca5f62-df90-42b8-91cb-dadacd5b3668-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7\" (UID: \"82ca5f62-df90-42b8-91cb-dadacd5b3668\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"
Oct 30 05:00:00.664649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664538    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtnw6\" (UniqueName: \"kubernetes.io/projected/06e8c6d6-1834-4ac0-a21d-998da69a4379-kube-api-access-dtnw6\") pod \"maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h\" (UID: \"06e8c6d6-1834-4ac0-a21d-998da69a4379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"
Oct 30 05:00:00.664649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664555    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f638f03d-08bb-4937-aa89-2198ac4a4c02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g\" (UID: \"f638f03d-08bb-4937-aa89-2198ac4a4c02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"
Oct 30 05:00:00.664649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664612    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e8c6d6-1834-4ac0-a21d-998da69a4379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h\" (UID: \"06e8c6d6-1834-4ac0-a21d-998da69a4379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"
Oct 30 05:00:00.664649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664644    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt\" (UID: \"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"
Oct 30 05:00:00.664836 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664665    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9acd468-3d02-4521-9db7-33e9dff0259f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs\" (UID: \"c9acd468-3d02-4521-9db7-33e9dff0259f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"
Oct 30 05:00:00.664836 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664719    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8n6b\" (UniqueName: \"kubernetes.io/projected/82ca5f62-df90-42b8-91cb-dadacd5b3668-kube-api-access-j8n6b\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7\" (UID: \"82ca5f62-df90-42b8-91cb-dadacd5b3668\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"
Oct 30 05:00:00.664836 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664747    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcvhb\" (UniqueName: \"kubernetes.io/projected/c9acd468-3d02-4521-9db7-33e9dff0259f-kube-api-access-vcvhb\") pod \"maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs\" (UID: \"c9acd468-3d02-4521-9db7-33e9dff0259f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"
Oct 30 05:00:00.664836 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664770    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmn5c\" (UniqueName: \"kubernetes.io/projected/7a540e59-8239-423a-ac80-aa84d7b4821d-kube-api-access-qmn5c\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw\" (UID: \"7a540e59-8239-423a-ac80-aa84d7b4821d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"
Oct 30 05:00:00.664836 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664787    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnvh6\" (UniqueName: \"kubernetes.io/projected/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-kube-api-access-pnvh6\") pod \"maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt\" (UID: \"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"
Oct 30 05:00:00.664836 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664819    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e846f7c-186a-4e49-ba15-6ff6c564bc13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6\" (UID: \"0e846f7c-186a-4e49-ba15-6ff6c564bc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"
Oct 30 05:00:00.665028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.664843    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a82ff41-ac42-48bc-81c4-24e539017282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf\" (UID: \"2a82ff41-ac42-48bc-81c4-24e539017282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"
Oct 30 05:00:00.665979 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.665930    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f638f03d-08bb-4937-aa89-2198ac4a4c02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g\" (UID: \"f638f03d-08bb-4937-aa89-2198ac4a4c02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"
Oct 30 05:00:00.666709 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.666680    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a82ff41-ac42-48bc-81c4-24e539017282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf\" (UID: \"2a82ff41-ac42-48bc-81c4-24e539017282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"
Oct 30 05:00:00.666806 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.666704    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt\" (UID: \"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"
Oct 30 05:00:00.667288 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.667268    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6f868c-5bea-49e5-a136-ab5862d1841d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c\" (UID: \"5d6f868c-5bea-49e5-a136-ab5862d1841d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"
Oct 30 05:00:00.667883 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.667845    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e846f7c-186a-4e49-ba15-6ff6c564bc13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6\" (UID: \"0e846f7c-186a-4e49-ba15-6ff6c564bc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"
Oct 30 05:00:00.668215 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.668193    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9acd468-3d02-4521-9db7-33e9dff0259f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs\" (UID: \"c9acd468-3d02-4521-9db7-33e9dff0259f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"
Oct 30 05:00:00.668372 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.668352    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc366c3b-6526-4575-899f-df191a5971de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2\" (UID: \"fc366c3b-6526-4575-899f-df191a5971de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"
Oct 30 05:00:00.668571 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.668550    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a540e59-8239-423a-ac80-aa84d7b4821d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw\" (UID: \"7a540e59-8239-423a-ac80-aa84d7b4821d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"
Oct 30 05:00:00.668913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.668890    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ca5f62-df90-42b8-91cb-dadacd5b3668-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7\" (UID: \"82ca5f62-df90-42b8-91cb-dadacd5b3668\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"
Oct 30 05:00:00.695411 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.695382    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgjj\" (UniqueName: \"kubernetes.io/projected/fc366c3b-6526-4575-899f-df191a5971de-kube-api-access-csgjj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2\" (UID: \"fc366c3b-6526-4575-899f-df191a5971de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"
Oct 30 05:00:00.697125 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.696984    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5b4f\" (UniqueName: \"kubernetes.io/projected/f638f03d-08bb-4937-aa89-2198ac4a4c02-kube-api-access-t5b4f\") pod \"maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g\" (UID: \"f638f03d-08bb-4937-aa89-2198ac4a4c02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"
Oct 30 05:00:00.700413 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.700372    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmn4h\" (UniqueName: \"kubernetes.io/projected/5d6f868c-5bea-49e5-a136-ab5862d1841d-kube-api-access-pmn4h\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c\" (UID: \"5d6f868c-5bea-49e5-a136-ab5862d1841d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"
Oct 30 05:00:00.700575 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.700548    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nlf\" (UniqueName: \"kubernetes.io/projected/2a82ff41-ac42-48bc-81c4-24e539017282-kube-api-access-84nlf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf\" (UID: \"2a82ff41-ac42-48bc-81c4-24e539017282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"
Oct 30 05:00:00.705721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.701921    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kd87\" (UniqueName: \"kubernetes.io/projected/0e846f7c-186a-4e49-ba15-6ff6c564bc13-kube-api-access-4kd87\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6\" (UID: \"0e846f7c-186a-4e49-ba15-6ff6c564bc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"
Oct 30 05:00:00.705721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.702137    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"
Oct 30 05:00:00.705935 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.705841    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmn5c\" (UniqueName: \"kubernetes.io/projected/7a540e59-8239-423a-ac80-aa84d7b4821d-kube-api-access-qmn5c\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw\" (UID: \"7a540e59-8239-423a-ac80-aa84d7b4821d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"
Oct 30 05:00:00.722967 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.722754    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"
Oct 30 05:00:00.724007 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.723949    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcvhb\" (UniqueName: \"kubernetes.io/projected/c9acd468-3d02-4521-9db7-33e9dff0259f-kube-api-access-vcvhb\") pod \"maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs\" (UID: \"c9acd468-3d02-4521-9db7-33e9dff0259f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"
Oct 30 05:00:00.726459 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.726330    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnvh6\" (UniqueName: \"kubernetes.io/projected/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-kube-api-access-pnvh6\") pod \"maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt\" (UID: \"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"
Oct 30 05:00:00.726459 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.726410    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8n6b\" (UniqueName: \"kubernetes.io/projected/82ca5f62-df90-42b8-91cb-dadacd5b3668-kube-api-access-j8n6b\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7\" (UID: \"82ca5f62-df90-42b8-91cb-dadacd5b3668\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"
Oct 30 05:00:00.734524 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.734470    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"
Oct 30 05:00:00.770605 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.770564    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtnw6\" (UniqueName: \"kubernetes.io/projected/06e8c6d6-1834-4ac0-a21d-998da69a4379-kube-api-access-dtnw6\") pod \"maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h\" (UID: \"06e8c6d6-1834-4ac0-a21d-998da69a4379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"
Oct 30 05:00:00.770760 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.770705    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e8c6d6-1834-4ac0-a21d-998da69a4379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h\" (UID: \"06e8c6d6-1834-4ac0-a21d-998da69a4379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"
Oct 30 05:00:00.771111 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.771058    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888940-bbnrh\" (UID: \"7fdb04a5-7116-41a3-80cd-f1169b75cc2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"
Oct 30 05:00:00.771219 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.771153    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68kcr\" (UniqueName: \"kubernetes.io/projected/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-kube-api-access-68kcr\") pod \"maas-api-key-cleanup-29888940-bbnrh\" (UID: \"7fdb04a5-7116-41a3-80cd-f1169b75cc2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"
Oct 30 05:00:00.772043 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.772011    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e8c6d6-1834-4ac0-a21d-998da69a4379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h\" (UID: \"06e8c6d6-1834-4ac0-a21d-998da69a4379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"
Oct 30 05:00:00.773208 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.773124    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888940-bbnrh\" (UID: \"7fdb04a5-7116-41a3-80cd-f1169b75cc2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"
Oct 30 05:00:00.790401 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.789069    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"
Oct 30 05:00:00.794798 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.794024    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"
Oct 30 05:00:00.843233 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.842707    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68kcr\" (UniqueName: \"kubernetes.io/projected/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-kube-api-access-68kcr\") pod \"maas-api-key-cleanup-29888940-bbnrh\" (UID: \"7fdb04a5-7116-41a3-80cd-f1169b75cc2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"
Oct 30 05:00:00.843616 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.843570    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtnw6\" (UniqueName: \"kubernetes.io/projected/06e8c6d6-1834-4ac0-a21d-998da69a4379-kube-api-access-dtnw6\") pod \"maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h\" (UID: \"06e8c6d6-1834-4ac0-a21d-998da69a4379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"
Oct 30 05:00:00.843759 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.843721    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"]
Oct 30 05:00:00.849378 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.849143    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"
Oct 30 05:00:00.855560 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.855358    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"
Oct 30 05:00:00.893543 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.886065    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"
Oct 30 05:00:00.893543 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.890434    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"
Oct 30 05:00:00.902115 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.901932    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"
Oct 30 05:00:00.903805 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.903727    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"
Oct 30 05:00:00.946821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.946784    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"
Oct 30 05:00:00.946935 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.946913    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"
Oct 30 05:00:00.963218 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.963185    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"
Oct 30 05:00:00.985862 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.985571    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"
Oct 30 05:00:00.998837 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:00.994886    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"
Oct 30 05:00:01.122452 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.122408    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"]
Oct 30 05:00:01.187475 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.187395    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"]
Oct 30 05:00:01.197229 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.196232    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9" event={"ID":"6fa98861-f2c2-47aa-bb83-54ddf8c5c715","Type":"ContainerStarted","Data":"0a746d3413376925fca781dc2f81e969d66a33faf0f37f72de7336ceb5544004"}
Oct 30 05:00:01.239999 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.239960    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"]
Oct 30 05:00:01.375627 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.375549    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"]
Oct 30 05:00:01.385118 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.384984    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"]
Oct 30 05:00:01.391679 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:01.391617    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53d6987_db5e_468a_9afe_2c0603cbf408.slice/crio-279813d32e6685eb916520774045eeed5079cfd5511059b430ff1f7a23502b72 WatchSource:0}: Error finding container 279813d32e6685eb916520774045eeed5079cfd5511059b430ff1f7a23502b72: Status 404 returned error can't find the container with id 279813d32e6685eb916520774045eeed5079cfd5511059b430ff1f7a23502b72
Oct 30 05:00:01.480531 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.480499    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"]
Oct 30 05:00:01.567141 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.567112    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"]
Oct 30 05:00:01.567898 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:01.567863    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5cb00e5_a10c_4e5a_846b_7e10e452611e.slice/crio-4d9d04323067c7a73d13a2dc9d6076e5c5ff44becc42fed97ef596cb6a69679b WatchSource:0}: Error finding container 4d9d04323067c7a73d13a2dc9d6076e5c5ff44becc42fed97ef596cb6a69679b: Status 404 returned error can't find the container with id 4d9d04323067c7a73d13a2dc9d6076e5c5ff44becc42fed97ef596cb6a69679b
Oct 30 05:00:01.568963 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:01.568934    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927eaf22_dd8b_4bef_a48c_fff361ead463.slice/crio-997d984231489801a631803e20c170d6d021780511401aae7da3679fc5d5ca12 WatchSource:0}: Error finding container 997d984231489801a631803e20c170d6d021780511401aae7da3679fc5d5ca12: Status 404 returned error can't find the container with id 997d984231489801a631803e20c170d6d021780511401aae7da3679fc5d5ca12
Oct 30 05:00:01.593888 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.593851    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"]
Oct 30 05:00:01.599295 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.599265    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"]
Oct 30 05:00:01.715374 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.715345    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"]
Oct 30 05:00:01.722762 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:01.722717    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7fc8fe_4ea2_4e79_9e10_fd0994f364e3.slice/crio-b3eea4e83e34a7a83019942e85f085dc25781e03a08e5ead06ec1722622d1f62 WatchSource:0}: Error finding container b3eea4e83e34a7a83019942e85f085dc25781e03a08e5ead06ec1722622d1f62: Status 404 returned error can't find the container with id b3eea4e83e34a7a83019942e85f085dc25781e03a08e5ead06ec1722622d1f62
Oct 30 05:00:01.740013 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.739983    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"]
Oct 30 05:00:01.743553 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:01.743528    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9acd468_3d02_4521_9db7_33e9dff0259f.slice/crio-9f0fd6ad2263913994c11e943071e1d6d7b5e48a4d91cb41c6d8910ada16d79a WatchSource:0}: Error finding container 9f0fd6ad2263913994c11e943071e1d6d7b5e48a4d91cb41c6d8910ada16d79a: Status 404 returned error can't find the container with id 9f0fd6ad2263913994c11e943071e1d6d7b5e48a4d91cb41c6d8910ada16d79a
Oct 30 05:00:01.814919 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.814893    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"]
Oct 30 05:00:01.830449 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:01.830421    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc366c3b_6526_4575_899f_df191a5971de.slice/crio-6fc771849ccab5fc3684ae8b2ee962160938999a54111927e20717d3c12fe9e8 WatchSource:0}: Error finding container 6fc771849ccab5fc3684ae8b2ee962160938999a54111927e20717d3c12fe9e8: Status 404 returned error can't find the container with id 6fc771849ccab5fc3684ae8b2ee962160938999a54111927e20717d3c12fe9e8
Oct 30 05:00:01.837022 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.836993    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"]
Oct 30 05:00:01.849943 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.849908    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"]
Oct 30 05:00:01.868855 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:01.866616    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ca5f62_df90_42b8_91cb_dadacd5b3668.slice/crio-9d20624bc36d147d343ea69c5a081fa0116887d95b85c6c40e6103d6066fa39c WatchSource:0}: Error finding container 9d20624bc36d147d343ea69c5a081fa0116887d95b85c6c40e6103d6066fa39c: Status 404 returned error can't find the container with id 9d20624bc36d147d343ea69c5a081fa0116887d95b85c6c40e6103d6066fa39c
Oct 30 05:00:01.875717 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.875691    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"]
Oct 30 05:00:01.885806 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.885775    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"]
Oct 30 05:00:01.950241 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.950207    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"]
Oct 30 05:00:01.986348 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:01.986306    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"]
Oct 30 05:00:02.002411 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.002377    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"]
Oct 30 05:00:02.042807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.042777    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"]
Oct 30 05:00:02.047311 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.047288    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"]
Oct 30 05:00:02.161795 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:02.161751    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf638f03d_08bb_4937_aa89_2198ac4a4c02.slice/crio-60a0c86cfadb947410136dbfb944ae60a0a5f93cd3e22d87887bd334055314ff WatchSource:0}: Error finding container 60a0c86cfadb947410136dbfb944ae60a0a5f93cd3e22d87887bd334055314ff: Status 404 returned error can't find the container with id 60a0c86cfadb947410136dbfb944ae60a0a5f93cd3e22d87887bd334055314ff
Oct 30 05:00:02.206517 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.206488    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n" event={"ID":"3621a9d6-ea49-415f-a6a9-5ce99245d111","Type":"ContainerStarted","Data":"138210d3db7aca72a22226d1504ba5759eee3e6abc5dd8a3f3b79281dc8b57ec"}
Oct 30 05:00:02.206906 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.206544    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n" event={"ID":"3621a9d6-ea49-415f-a6a9-5ce99245d111","Type":"ContainerStarted","Data":"686f7959bda36c40185c3272dfa8cf20ada2bc1e9bb4ae66ec5c6a28ea31441c"}
Oct 30 05:00:02.207893 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.207871    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2" event={"ID":"fc366c3b-6526-4575-899f-df191a5971de","Type":"ContainerStarted","Data":"6fc771849ccab5fc3684ae8b2ee962160938999a54111927e20717d3c12fe9e8"}
Oct 30 05:00:02.209128 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.209106    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw" event={"ID":"d8802708-e588-476f-b55b-bce1a3efa0cd","Type":"ContainerStarted","Data":"788283d500347b6558b8e91370440c725f2ca322453e7f3ec4ed13bc3c56f175"}
Oct 30 05:00:02.210399 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.210378    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk" event={"ID":"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61","Type":"ContainerStarted","Data":"6cd971112f1392c75fdf8cee40058edadc236f1c1e76b8e66579d0bea28144e8"}
Oct 30 05:00:02.211666 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.211645    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb" event={"ID":"92e82ab9-06b8-4c71-b46a-76348dee31b5","Type":"ContainerStarted","Data":"1f54b7c8c4033eae462f21c1acb4efb72aeab038f0effc70bb2e8b2c20b55dc9"}
Oct 30 05:00:02.212886 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.212865    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c" event={"ID":"5d6f868c-5bea-49e5-a136-ab5862d1841d","Type":"ContainerStarted","Data":"faa2a3a9ad07d050e6ae9b57e4e2f1fc973207a37d117b36a52d919131b20b1b"}
Oct 30 05:00:02.214130 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.214112    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs" event={"ID":"c9acd468-3d02-4521-9db7-33e9dff0259f","Type":"ContainerStarted","Data":"9f0fd6ad2263913994c11e943071e1d6d7b5e48a4d91cb41c6d8910ada16d79a"}
Oct 30 05:00:02.215459 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.215439    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g" event={"ID":"f638f03d-08bb-4937-aa89-2198ac4a4c02","Type":"ContainerStarted","Data":"60a0c86cfadb947410136dbfb944ae60a0a5f93cd3e22d87887bd334055314ff"}
Oct 30 05:00:02.216698 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.216677    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7" event={"ID":"82ca5f62-df90-42b8-91cb-dadacd5b3668","Type":"ContainerStarted","Data":"9d20624bc36d147d343ea69c5a081fa0116887d95b85c6c40e6103d6066fa39c"}
Oct 30 05:00:02.217918 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.217899    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h" event={"ID":"804abcc6-7570-4f94-be22-d2c8b12f2108","Type":"ContainerStarted","Data":"79d07a4e826d98be25f445ac78462cf98efa878e24fd85be3b9c0dbffd835534"}
Oct 30 05:00:02.219198 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.219174    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d" event={"ID":"a5cb00e5-a10c-4e5a-846b-7e10e452611e","Type":"ContainerStarted","Data":"4d9d04323067c7a73d13a2dc9d6076e5c5ff44becc42fed97ef596cb6a69679b"}
Oct 30 05:00:02.220368 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.220350    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh" event={"ID":"7fdb04a5-7116-41a3-80cd-f1169b75cc2b","Type":"ContainerStarted","Data":"f799aa5f529902b321693e149153ab2e44b5b164ad90890d30946cbf05261bf3"}
Oct 30 05:00:02.221601 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.221581    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt" event={"ID":"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3","Type":"ContainerStarted","Data":"b3eea4e83e34a7a83019942e85f085dc25781e03a08e5ead06ec1722622d1f62"}
Oct 30 05:00:02.222908 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.222888    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv" event={"ID":"f52b50b1-dfe7-4316-9918-61d44366cd60","Type":"ContainerStarted","Data":"818d70685dc12f54d10f0a41924ce014863b0a5cde881dd6a2385e1f98aa665c"}
Oct 30 05:00:02.224119 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.224097    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9" event={"ID":"212e51ce-0a69-4809-822f-6f6fe6bf8773","Type":"ContainerStarted","Data":"c5ff0fcb8f36279fe3dad03a77915c7ad8c5ae4f7704c7f26f1169a916fd2122"}
Oct 30 05:00:02.225360 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.225338    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" event={"ID":"0e846f7c-186a-4e49-ba15-6ff6c564bc13","Type":"ContainerStarted","Data":"40006da4867c7690362e750a151751eaa6cc65e45baf581a0a9ffe5b6bd9311e"}
Oct 30 05:00:02.226816 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.226788    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9" event={"ID":"6fa98861-f2c2-47aa-bb83-54ddf8c5c715","Type":"ContainerStarted","Data":"602ad394c87089f452b930f3b94b0c108fd515484d260a700ff8f254dc947814"}
Oct 30 05:00:02.228551 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.228529    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5" event={"ID":"e53d6987-db5e-468a-9afe-2c0603cbf408","Type":"ContainerStarted","Data":"279813d32e6685eb916520774045eeed5079cfd5511059b430ff1f7a23502b72"}
Oct 30 05:00:02.229988 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.229969    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m" event={"ID":"927eaf22-dd8b-4bef-a48c-fff361ead463","Type":"ContainerStarted","Data":"997d984231489801a631803e20c170d6d021780511401aae7da3679fc5d5ca12"}
Oct 30 05:00:02.250008 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.249917    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n" podStartSLOduration=2.24990351 podStartE2EDuration="2.24990351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:02.248643421 +0000 UTC m=+3781.990066519" watchObservedRunningTime="2026-10-30 05:00:02.24990351 +0000 UTC m=+3781.991326607"
Oct 30 05:00:02.264672 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:02.264640    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e8c6d6_1834_4ac0_a21d_998da69a4379.slice/crio-f131b2276fbfa62ce6f1a70d029fecdc57e56d4a0b8c2ab160cc3be03fceb437 WatchSource:0}: Error finding container f131b2276fbfa62ce6f1a70d029fecdc57e56d4a0b8c2ab160cc3be03fceb437: Status 404 returned error can't find the container with id f131b2276fbfa62ce6f1a70d029fecdc57e56d4a0b8c2ab160cc3be03fceb437
Oct 30 05:00:02.295552 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:02.295497    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9" podStartSLOduration=2.295483883 podStartE2EDuration="2.295483883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:02.289764224 +0000 UTC m=+3782.031187322" watchObservedRunningTime="2026-10-30 05:00:02.295483883 +0000 UTC m=+3782.036906979"
Oct 30 05:00:02.360285 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:02.360257    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a82ff41_ac42_48bc_81c4_24e539017282.slice/crio-56ea82144e74659d1b4657ae9478cf31c65b71a76f52677d364121081f571fa1 WatchSource:0}: Error finding container 56ea82144e74659d1b4657ae9478cf31c65b71a76f52677d364121081f571fa1: Status 404 returned error can't find the container with id 56ea82144e74659d1b4657ae9478cf31c65b71a76f52677d364121081f571fa1
Oct 30 05:00:02.460574 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:02.460546    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a540e59_8239_423a_ac80_aa84d7b4821d.slice/crio-5422d3478931f9532833882e65891fc2e3b4636600cac4206891a9855f019b3b WatchSource:0}: Error finding container 5422d3478931f9532833882e65891fc2e3b4636600cac4206891a9855f019b3b: Status 404 returned error can't find the container with id 5422d3478931f9532833882e65891fc2e3b4636600cac4206891a9855f019b3b
Oct 30 05:00:03.240799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.240767    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw" event={"ID":"d8802708-e588-476f-b55b-bce1a3efa0cd","Type":"ContainerStarted","Data":"d3f642956f0df5de2f96de63ba433193c832d0489c44ee490623b112e9f89a9f"}
Oct 30 05:00:03.242781 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.242754    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk" event={"ID":"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61","Type":"ContainerStarted","Data":"9930286fdd71012fa66ad8a32e39a8d6cbf8beaabdb66b2fbee6ac30e773ea20"}
Oct 30 05:00:03.244576 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.244552    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb" event={"ID":"92e82ab9-06b8-4c71-b46a-76348dee31b5","Type":"ContainerStarted","Data":"392e05a495e6d8bb7bc2ddcd802e20e7f1c80ed65220b590b172ee786faa5eb1"}
Oct 30 05:00:03.246432 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.246402    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs" event={"ID":"c9acd468-3d02-4521-9db7-33e9dff0259f","Type":"ContainerStarted","Data":"608924cc655a3d257c4bf7d6dbbbfb97a51553b28a9e2cbf9cd55dcc7f3c9429"}
Oct 30 05:00:03.247877 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.247854    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h" event={"ID":"804abcc6-7570-4f94-be22-d2c8b12f2108","Type":"ContainerStarted","Data":"194a5e992ac1667fbfc3318ee18418fda85740a6e129f772d7b1f95fa7b2848f"}
Oct 30 05:00:03.249721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.249695    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d" event={"ID":"a5cb00e5-a10c-4e5a-846b-7e10e452611e","Type":"ContainerStarted","Data":"757a59225b257b8a0696291740cb104a3cf9da536d54f0996ee2cd7b30b2e402"}
Oct 30 05:00:03.251528 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.251508    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt" event={"ID":"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3","Type":"ContainerStarted","Data":"9dbaa42c4382ff1c0d6fe092a4d0a271faa9b0a66fd741afe36b939a9162fac0"}
Oct 30 05:00:03.253390 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.253363    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv" event={"ID":"f52b50b1-dfe7-4316-9918-61d44366cd60","Type":"ContainerStarted","Data":"e669a446601af077799f9626ed1d05d829bc8730f72a6ba65926c5daa0310cc6"}
Oct 30 05:00:03.254977 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.254956    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" event={"ID":"0e846f7c-186a-4e49-ba15-6ff6c564bc13","Type":"ContainerStarted","Data":"a6f45048bbf003aff2dae5aeff52f7fe751feac63ade2dbeb1ff74757b8c1bbf"}
Oct 30 05:00:03.257734 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.257703    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h" event={"ID":"06e8c6d6-1834-4ac0-a21d-998da69a4379","Type":"ContainerStarted","Data":"f131b2276fbfa62ce6f1a70d029fecdc57e56d4a0b8c2ab160cc3be03fceb437"}
Oct 30 05:00:03.259253 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.259231    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf" event={"ID":"2a82ff41-ac42-48bc-81c4-24e539017282","Type":"ContainerStarted","Data":"56ea82144e74659d1b4657ae9478cf31c65b71a76f52677d364121081f571fa1"}
Oct 30 05:00:03.264547 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.263886    5072 generic.go:356] "Generic (PLEG): container finished" podID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" containerID="602ad394c87089f452b930f3b94b0c108fd515484d260a700ff8f254dc947814" exitCode=60
Oct 30 05:00:03.264547 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.264047    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9" event={"ID":"6fa98861-f2c2-47aa-bb83-54ddf8c5c715","Type":"ContainerDied","Data":"602ad394c87089f452b930f3b94b0c108fd515484d260a700ff8f254dc947814"}
Oct 30 05:00:03.266299 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.266153    5072 scope.go:122] "RemoveContainer" containerID="602ad394c87089f452b930f3b94b0c108fd515484d260a700ff8f254dc947814"
Oct 30 05:00:03.268242 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.268215    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5" event={"ID":"e53d6987-db5e-468a-9afe-2c0603cbf408","Type":"ContainerStarted","Data":"950263a077853fffb7d0f94d647544951e930c7bf204aff3a8d8caed3c4ca95d"}
Oct 30 05:00:03.270959 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.270907    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw" event={"ID":"7a540e59-8239-423a-ac80-aa84d7b4821d","Type":"ContainerStarted","Data":"68e4bfd767bd9c8e62cfd6f87426c64ce8dd18b4274a60d76077afeccdc79ee8"}
Oct 30 05:00:03.271055 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.271036    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw" event={"ID":"7a540e59-8239-423a-ac80-aa84d7b4821d","Type":"ContainerStarted","Data":"5422d3478931f9532833882e65891fc2e3b4636600cac4206891a9855f019b3b"}
Oct 30 05:00:03.278142 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.276811    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m" event={"ID":"927eaf22-dd8b-4bef-a48c-fff361ead463","Type":"ContainerStarted","Data":"c22f88016d803962f88d3075c5a08c238d83060e2684dfd7486ff066832dcf83"}
Oct 30 05:00:03.279067 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.279039    5072 generic.go:356] "Generic (PLEG): container finished" podID="3621a9d6-ea49-415f-a6a9-5ce99245d111" containerID="138210d3db7aca72a22226d1504ba5759eee3e6abc5dd8a3f3b79281dc8b57ec" exitCode=60
Oct 30 05:00:03.279478 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.279213    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n" event={"ID":"3621a9d6-ea49-415f-a6a9-5ce99245d111","Type":"ContainerDied","Data":"138210d3db7aca72a22226d1504ba5759eee3e6abc5dd8a3f3b79281dc8b57ec"}
Oct 30 05:00:03.279734 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.279716    5072 scope.go:122] "RemoveContainer" containerID="138210d3db7aca72a22226d1504ba5759eee3e6abc5dd8a3f3b79281dc8b57ec"
Oct 30 05:00:03.295033 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.294991    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw" podStartSLOduration=3.294979078 podStartE2EDuration="3.294979078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.287278178 +0000 UTC m=+3783.028701275" watchObservedRunningTime="2026-10-30 05:00:03.294979078 +0000 UTC m=+3783.036402169"
Oct 30 05:00:03.493479 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.493425    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv" podStartSLOduration=3.49340925 podStartE2EDuration="3.49340925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.42774299 +0000 UTC m=+3783.169166087" watchObservedRunningTime="2026-10-30 05:00:03.49340925 +0000 UTC m=+3783.234832357"
Oct 30 05:00:03.495847 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.495800    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h" podStartSLOduration=3.495783319 podStartE2EDuration="3.495783319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.468194129 +0000 UTC m=+3783.209617229" watchObservedRunningTime="2026-10-30 05:00:03.495783319 +0000 UTC m=+3783.237206416"
Oct 30 05:00:03.510365 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.510317    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" podStartSLOduration=3.510300604 podStartE2EDuration="3.510300604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.509735296 +0000 UTC m=+3783.251158395" watchObservedRunningTime="2026-10-30 05:00:03.510300604 +0000 UTC m=+3783.251723702"
Oct 30 05:00:03.560228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.555161    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m" podStartSLOduration=3.555147948 podStartE2EDuration="3.555147948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.554737382 +0000 UTC m=+3783.296160479" watchObservedRunningTime="2026-10-30 05:00:03.555147948 +0000 UTC m=+3783.296571060"
Oct 30 05:00:03.600350 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.600304    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d" podStartSLOduration=3.600290824 podStartE2EDuration="3.600290824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.598516936 +0000 UTC m=+3783.339940034" watchObservedRunningTime="2026-10-30 05:00:03.600290824 +0000 UTC m=+3783.341713920"
Oct 30 05:00:03.695194 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.695136    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw" podStartSLOduration=3.695115927 podStartE2EDuration="3.695115927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.648324576 +0000 UTC m=+3783.389747674" watchObservedRunningTime="2026-10-30 05:00:03.695115927 +0000 UTC m=+3783.436539025"
Oct 30 05:00:03.696346 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.696297    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs" podStartSLOduration=3.696283645 podStartE2EDuration="3.696283645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.692510687 +0000 UTC m=+3783.433933784" watchObservedRunningTime="2026-10-30 05:00:03.696283645 +0000 UTC m=+3783.437706745"
Oct 30 05:00:03.737913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.737874    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt" podStartSLOduration=3.737862693 podStartE2EDuration="3.737862693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.736973572 +0000 UTC m=+3783.478396668" watchObservedRunningTime="2026-10-30 05:00:03.737862693 +0000 UTC m=+3783.479285789"
Oct 30 05:00:03.798715 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.798636    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb" podStartSLOduration=3.798615019 podStartE2EDuration="3.798615019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.788674608 +0000 UTC m=+3783.530097705" watchObservedRunningTime="2026-10-30 05:00:03.798615019 +0000 UTC m=+3783.540038118"
Oct 30 05:00:03.839358 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.839294    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk" podStartSLOduration=3.839273107 podStartE2EDuration="3.839273107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.834643798 +0000 UTC m=+3783.576066896" watchObservedRunningTime="2026-10-30 05:00:03.839273107 +0000 UTC m=+3783.580696208"
Oct 30 05:00:03.885963 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:03.885867    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5" podStartSLOduration=3.885847734 podStartE2EDuration="3.885847734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.879841232 +0000 UTC m=+3783.621264330" watchObservedRunningTime="2026-10-30 05:00:03.885847734 +0000 UTC m=+3783.627270832"
Oct 30 05:00:04.290445 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.290239    5072 generic.go:356] "Generic (PLEG): container finished" podID="e53d6987-db5e-468a-9afe-2c0603cbf408" containerID="950263a077853fffb7d0f94d647544951e930c7bf204aff3a8d8caed3c4ca95d" exitCode=60
Oct 30 05:00:04.290445 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.290331    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5" event={"ID":"e53d6987-db5e-468a-9afe-2c0603cbf408","Type":"ContainerDied","Data":"950263a077853fffb7d0f94d647544951e930c7bf204aff3a8d8caed3c4ca95d"}
Oct 30 05:00:04.290876 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.290741    5072 scope.go:122] "RemoveContainer" containerID="950263a077853fffb7d0f94d647544951e930c7bf204aff3a8d8caed3c4ca95d"
Oct 30 05:00:04.292491 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.292465    5072 generic.go:356] "Generic (PLEG): container finished" podID="7a540e59-8239-423a-ac80-aa84d7b4821d" containerID="68e4bfd767bd9c8e62cfd6f87426c64ce8dd18b4274a60d76077afeccdc79ee8" exitCode=60
Oct 30 05:00:04.292607 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.292561    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw" event={"ID":"7a540e59-8239-423a-ac80-aa84d7b4821d","Type":"ContainerDied","Data":"68e4bfd767bd9c8e62cfd6f87426c64ce8dd18b4274a60d76077afeccdc79ee8"}
Oct 30 05:00:04.292813 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.292795    5072 scope.go:122] "RemoveContainer" containerID="68e4bfd767bd9c8e62cfd6f87426c64ce8dd18b4274a60d76077afeccdc79ee8"
Oct 30 05:00:04.294704 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.294683    5072 generic.go:356] "Generic (PLEG): container finished" podID="927eaf22-dd8b-4bef-a48c-fff361ead463" containerID="c22f88016d803962f88d3075c5a08c238d83060e2684dfd7486ff066832dcf83" exitCode=60
Oct 30 05:00:04.294817 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.294792    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m" event={"ID":"927eaf22-dd8b-4bef-a48c-fff361ead463","Type":"ContainerDied","Data":"c22f88016d803962f88d3075c5a08c238d83060e2684dfd7486ff066832dcf83"}
Oct 30 05:00:04.295031 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.295012    5072 scope.go:122] "RemoveContainer" containerID="c22f88016d803962f88d3075c5a08c238d83060e2684dfd7486ff066832dcf83"
Oct 30 05:00:04.296933 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.296858    5072 generic.go:356] "Generic (PLEG): container finished" podID="fc366c3b-6526-4575-899f-df191a5971de" containerID="6f52ec0b06275a65c38aa66fa8e4430e3c4c538cf49e9a299b4f1eeb73b9160a" exitCode=60
Oct 30 05:00:04.296996 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.296944    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2" event={"ID":"fc366c3b-6526-4575-899f-df191a5971de","Type":"ContainerDied","Data":"6f52ec0b06275a65c38aa66fa8e4430e3c4c538cf49e9a299b4f1eeb73b9160a"}
Oct 30 05:00:04.297295 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.297276    5072 scope.go:122] "RemoveContainer" containerID="6f52ec0b06275a65c38aa66fa8e4430e3c4c538cf49e9a299b4f1eeb73b9160a"
Oct 30 05:00:04.298724 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.298704    5072 generic.go:356] "Generic (PLEG): container finished" podID="d8802708-e588-476f-b55b-bce1a3efa0cd" containerID="d3f642956f0df5de2f96de63ba433193c832d0489c44ee490623b112e9f89a9f" exitCode=60
Oct 30 05:00:04.298831 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.298785    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw" event={"ID":"d8802708-e588-476f-b55b-bce1a3efa0cd","Type":"ContainerDied","Data":"d3f642956f0df5de2f96de63ba433193c832d0489c44ee490623b112e9f89a9f"}
Oct 30 05:00:04.298982 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.298966    5072 scope.go:122] "RemoveContainer" containerID="d3f642956f0df5de2f96de63ba433193c832d0489c44ee490623b112e9f89a9f"
Oct 30 05:00:04.300796 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.300775    5072 generic.go:356] "Generic (PLEG): container finished" podID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" containerID="9930286fdd71012fa66ad8a32e39a8d6cbf8beaabdb66b2fbee6ac30e773ea20" exitCode=60
Oct 30 05:00:04.300882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.300853    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk" event={"ID":"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61","Type":"ContainerDied","Data":"9930286fdd71012fa66ad8a32e39a8d6cbf8beaabdb66b2fbee6ac30e773ea20"}
Oct 30 05:00:04.301036 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.301023    5072 scope.go:122] "RemoveContainer" containerID="9930286fdd71012fa66ad8a32e39a8d6cbf8beaabdb66b2fbee6ac30e773ea20"
Oct 30 05:00:04.302623 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.302601    5072 generic.go:356] "Generic (PLEG): container finished" podID="92e82ab9-06b8-4c71-b46a-76348dee31b5" containerID="392e05a495e6d8bb7bc2ddcd802e20e7f1c80ed65220b590b172ee786faa5eb1" exitCode=60
Oct 30 05:00:04.302696 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.302625    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb" event={"ID":"92e82ab9-06b8-4c71-b46a-76348dee31b5","Type":"ContainerDied","Data":"392e05a495e6d8bb7bc2ddcd802e20e7f1c80ed65220b590b172ee786faa5eb1"}
Oct 30 05:00:04.302853 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.302840    5072 scope.go:122] "RemoveContainer" containerID="392e05a495e6d8bb7bc2ddcd802e20e7f1c80ed65220b590b172ee786faa5eb1"
Oct 30 05:00:04.304263 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.304245    5072 generic.go:356] "Generic (PLEG): container finished" podID="5d6f868c-5bea-49e5-a136-ab5862d1841d" containerID="e042f4d711102e125592724cbbb8dedbcb0d2ef58a7294e6cefe939a6edc103c" exitCode=60
Oct 30 05:00:04.304351 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.304304    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c" event={"ID":"5d6f868c-5bea-49e5-a136-ab5862d1841d","Type":"ContainerDied","Data":"e042f4d711102e125592724cbbb8dedbcb0d2ef58a7294e6cefe939a6edc103c"}
Oct 30 05:00:04.304513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.304498    5072 scope.go:122] "RemoveContainer" containerID="e042f4d711102e125592724cbbb8dedbcb0d2ef58a7294e6cefe939a6edc103c"
Oct 30 05:00:04.307283 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.307002    5072 generic.go:356] "Generic (PLEG): container finished" podID="c9acd468-3d02-4521-9db7-33e9dff0259f" containerID="608924cc655a3d257c4bf7d6dbbbfb97a51553b28a9e2cbf9cd55dcc7f3c9429" exitCode=60
Oct 30 05:00:04.307283 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.307122    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs" event={"ID":"c9acd468-3d02-4521-9db7-33e9dff0259f","Type":"ContainerDied","Data":"608924cc655a3d257c4bf7d6dbbbfb97a51553b28a9e2cbf9cd55dcc7f3c9429"}
Oct 30 05:00:04.309548 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.308741    5072 scope.go:122] "RemoveContainer" containerID="608924cc655a3d257c4bf7d6dbbbfb97a51553b28a9e2cbf9cd55dcc7f3c9429"
Oct 30 05:00:04.310636 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.310616    5072 generic.go:356] "Generic (PLEG): container finished" podID="f638f03d-08bb-4937-aa89-2198ac4a4c02" containerID="f841a7f5b7bb8cc5b63475258a56e29340904242854275b6c8ddf25ba98d99e7" exitCode=60
Oct 30 05:00:04.310762 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.310673    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g" event={"ID":"f638f03d-08bb-4937-aa89-2198ac4a4c02","Type":"ContainerDied","Data":"f841a7f5b7bb8cc5b63475258a56e29340904242854275b6c8ddf25ba98d99e7"}
Oct 30 05:00:04.310897 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.310880    5072 scope.go:122] "RemoveContainer" containerID="f841a7f5b7bb8cc5b63475258a56e29340904242854275b6c8ddf25ba98d99e7"
Oct 30 05:00:04.312661 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.312641    5072 generic.go:356] "Generic (PLEG): container finished" podID="82ca5f62-df90-42b8-91cb-dadacd5b3668" containerID="9cc782b6bca432cf512cba6ffcbccb9c6481bab8baf9b3c77a729b98196e2abd" exitCode=60
Oct 30 05:00:04.312762 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.312713    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7" event={"ID":"82ca5f62-df90-42b8-91cb-dadacd5b3668","Type":"ContainerDied","Data":"9cc782b6bca432cf512cba6ffcbccb9c6481bab8baf9b3c77a729b98196e2abd"}
Oct 30 05:00:04.312932 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.312910    5072 scope.go:122] "RemoveContainer" containerID="9cc782b6bca432cf512cba6ffcbccb9c6481bab8baf9b3c77a729b98196e2abd"
Oct 30 05:00:04.314566 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.314542    5072 generic.go:356] "Generic (PLEG): container finished" podID="804abcc6-7570-4f94-be22-d2c8b12f2108" containerID="194a5e992ac1667fbfc3318ee18418fda85740a6e129f772d7b1f95fa7b2848f" exitCode=60
Oct 30 05:00:04.314649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.314616    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h" event={"ID":"804abcc6-7570-4f94-be22-d2c8b12f2108","Type":"ContainerDied","Data":"194a5e992ac1667fbfc3318ee18418fda85740a6e129f772d7b1f95fa7b2848f"}
Oct 30 05:00:04.314803 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.314788    5072 scope.go:122] "RemoveContainer" containerID="194a5e992ac1667fbfc3318ee18418fda85740a6e129f772d7b1f95fa7b2848f"
Oct 30 05:00:04.317372 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.317351    5072 generic.go:356] "Generic (PLEG): container finished" podID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" containerID="757a59225b257b8a0696291740cb104a3cf9da536d54f0996ee2cd7b30b2e402" exitCode=60
Oct 30 05:00:04.317883 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.317439    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d" event={"ID":"a5cb00e5-a10c-4e5a-846b-7e10e452611e","Type":"ContainerDied","Data":"757a59225b257b8a0696291740cb104a3cf9da536d54f0996ee2cd7b30b2e402"}
Oct 30 05:00:04.317883 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.317683    5072 scope.go:122] "RemoveContainer" containerID="757a59225b257b8a0696291740cb104a3cf9da536d54f0996ee2cd7b30b2e402"
Oct 30 05:00:04.321868 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.320279    5072 generic.go:356] "Generic (PLEG): container finished" podID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" containerID="dace9df966feb7f31457c1c803304e48fd5dc4b550944dd5f51d185dd5bf5186" exitCode=60
Oct 30 05:00:04.321868 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.320341    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh" event={"ID":"7fdb04a5-7116-41a3-80cd-f1169b75cc2b","Type":"ContainerDied","Data":"dace9df966feb7f31457c1c803304e48fd5dc4b550944dd5f51d185dd5bf5186"}
Oct 30 05:00:04.321868 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.320657    5072 scope.go:122] "RemoveContainer" containerID="dace9df966feb7f31457c1c803304e48fd5dc4b550944dd5f51d185dd5bf5186"
Oct 30 05:00:04.322906 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.322881    5072 generic.go:356] "Generic (PLEG): container finished" podID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" containerID="9dbaa42c4382ff1c0d6fe092a4d0a271faa9b0a66fd741afe36b939a9162fac0" exitCode=60
Oct 30 05:00:04.322993 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.322964    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt" event={"ID":"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3","Type":"ContainerDied","Data":"9dbaa42c4382ff1c0d6fe092a4d0a271faa9b0a66fd741afe36b939a9162fac0"}
Oct 30 05:00:04.323325 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.323304    5072 scope.go:122] "RemoveContainer" containerID="9dbaa42c4382ff1c0d6fe092a4d0a271faa9b0a66fd741afe36b939a9162fac0"
Oct 30 05:00:04.326361 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.326337    5072 generic.go:356] "Generic (PLEG): container finished" podID="f52b50b1-dfe7-4316-9918-61d44366cd60" containerID="e669a446601af077799f9626ed1d05d829bc8730f72a6ba65926c5daa0310cc6" exitCode=60
Oct 30 05:00:04.326456 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.326387    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv" event={"ID":"f52b50b1-dfe7-4316-9918-61d44366cd60","Type":"ContainerDied","Data":"e669a446601af077799f9626ed1d05d829bc8730f72a6ba65926c5daa0310cc6"}
Oct 30 05:00:04.326661 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.326641    5072 scope.go:122] "RemoveContainer" containerID="e669a446601af077799f9626ed1d05d829bc8730f72a6ba65926c5daa0310cc6"
Oct 30 05:00:04.331455 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.331437    5072 generic.go:356] "Generic (PLEG): container finished" podID="212e51ce-0a69-4809-822f-6f6fe6bf8773" containerID="986a374e1101aca1ba3000fa3f5748453a2a561a108c440ea93937004ddbc2ef" exitCode=60
Oct 30 05:00:04.331532 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.331517    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9" event={"ID":"212e51ce-0a69-4809-822f-6f6fe6bf8773","Type":"ContainerDied","Data":"986a374e1101aca1ba3000fa3f5748453a2a561a108c440ea93937004ddbc2ef"}
Oct 30 05:00:04.332224 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.331795    5072 scope.go:122] "RemoveContainer" containerID="986a374e1101aca1ba3000fa3f5748453a2a561a108c440ea93937004ddbc2ef"
Oct 30 05:00:04.335203 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.334493    5072 generic.go:356] "Generic (PLEG): container finished" podID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" containerID="a6f45048bbf003aff2dae5aeff52f7fe751feac63ade2dbeb1ff74757b8c1bbf" exitCode=60
Oct 30 05:00:04.335203 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.334573    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" event={"ID":"0e846f7c-186a-4e49-ba15-6ff6c564bc13","Type":"ContainerDied","Data":"a6f45048bbf003aff2dae5aeff52f7fe751feac63ade2dbeb1ff74757b8c1bbf"}
Oct 30 05:00:04.343116 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.342128    5072 scope.go:122] "RemoveContainer" containerID="a6f45048bbf003aff2dae5aeff52f7fe751feac63ade2dbeb1ff74757b8c1bbf"
Oct 30 05:00:04.343116 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.342822    5072 generic.go:356] "Generic (PLEG): container finished" podID="06e8c6d6-1834-4ac0-a21d-998da69a4379" containerID="4528a89ba0b1bebd3cda0fdaf74c08f14bee541ede5f7a90d7283516917b001c" exitCode=60
Oct 30 05:00:04.343116 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.342902    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h" event={"ID":"06e8c6d6-1834-4ac0-a21d-998da69a4379","Type":"ContainerDied","Data":"4528a89ba0b1bebd3cda0fdaf74c08f14bee541ede5f7a90d7283516917b001c"}
Oct 30 05:00:04.343422 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.343237    5072 scope.go:122] "RemoveContainer" containerID="4528a89ba0b1bebd3cda0fdaf74c08f14bee541ede5f7a90d7283516917b001c"
Oct 30 05:00:04.347220 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.347198    5072 generic.go:356] "Generic (PLEG): container finished" podID="2a82ff41-ac42-48bc-81c4-24e539017282" containerID="f2ac90afd86c848d307a7a7c1a6d2716960c4802d3a44dd66ab09ee548c3da7d" exitCode=60
Oct 30 05:00:04.347306 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.347225    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf" event={"ID":"2a82ff41-ac42-48bc-81c4-24e539017282","Type":"ContainerDied","Data":"f2ac90afd86c848d307a7a7c1a6d2716960c4802d3a44dd66ab09ee548c3da7d"}
Oct 30 05:00:04.347534 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:04.347516    5072 scope.go:122] "RemoveContainer" containerID="f2ac90afd86c848d307a7a7c1a6d2716960c4802d3a44dd66ab09ee548c3da7d"
Oct 30 05:00:05.034712 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.034676    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"]
Oct 30 05:00:05.040140 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.040105    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2526b9d9-v8zp5"]
Oct 30 05:00:05.361575 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.361542    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c" event={"ID":"5d6f868c-5bea-49e5-a136-ab5862d1841d","Type":"ContainerStarted","Data":"3716493d7a1d9bf72fffa52bb393512bfbcc2eb222115fd3cbfe445f2e4a0457"}
Oct 30 05:00:05.366200 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.366168    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9" event={"ID":"6fa98861-f2c2-47aa-bb83-54ddf8c5c715","Type":"ContainerStarted","Data":"16e10d0b9c9bd56da65bf95af053716430b0dbf497d5bc33241e767f8b09bf49"}
Oct 30 05:00:05.369422 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.369394    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5" event={"ID":"e53d6987-db5e-468a-9afe-2c0603cbf408","Type":"ContainerStarted","Data":"8b48e76ffc3256ff64f90aaf9f650b41e6f353ebebc176b861874f746cb85f2b"}
Oct 30 05:00:05.371859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.371835    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw" event={"ID":"7a540e59-8239-423a-ac80-aa84d7b4821d","Type":"ContainerStarted","Data":"d365486c30671c65b714592a62e9ed7113fb64bddae0406abafdd0568eb4fbd5"}
Oct 30 05:00:05.374973 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.374718    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m" event={"ID":"927eaf22-dd8b-4bef-a48c-fff361ead463","Type":"ContainerStarted","Data":"11c3e6636e37e94b9b34a11c8ba06c213ec7881f2f8a9966e7afa509f06bf722"}
Oct 30 05:00:05.377274 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.377228    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n" event={"ID":"3621a9d6-ea49-415f-a6a9-5ce99245d111","Type":"ContainerStarted","Data":"591a426acee69061f04dd12a71dab7b97f9e69b1b4128d3de05b4312c743be24"}
Oct 30 05:00:05.379242 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.379215    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2" event={"ID":"fc366c3b-6526-4575-899f-df191a5971de","Type":"ContainerStarted","Data":"da2e437e983141ec02fa7b21035f90ccec5f0efd176f4dd790833fabb8902c5e"}
Oct 30 05:00:05.381299 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.381278    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw" event={"ID":"d8802708-e588-476f-b55b-bce1a3efa0cd","Type":"ContainerStarted","Data":"519099ca6ab822c89659ec20045ca80e7a3fcb5a0acb95987001449c13e8c014"}
Oct 30 05:00:05.383617 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.383596    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk" event={"ID":"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61","Type":"ContainerStarted","Data":"9e990a2dd0ebc27338d0b1fb4c8d60135554438ebc178741dba87e83fd8bdb39"}
Oct 30 05:00:05.385927 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.385899    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb" event={"ID":"92e82ab9-06b8-4c71-b46a-76348dee31b5","Type":"ContainerStarted","Data":"2138d1a10a60a072b9380a25ca7eafc48ad0145135688f95fbec566c89b5a11e"}
Oct 30 05:00:05.405165 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.405126    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c" podStartSLOduration=5.405113248 podStartE2EDuration="5.405113248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:05.403419366 +0000 UTC m=+3785.144842463" watchObservedRunningTime="2026-10-30 05:00:05.405113248 +0000 UTC m=+3785.146536345"
Oct 30 05:00:05.535171 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:05.535065    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2" podStartSLOduration=5.535049123 podStartE2EDuration="5.535049123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:05.533279892 +0000 UTC m=+3785.274702989" watchObservedRunningTime="2026-10-30 05:00:05.535049123 +0000 UTC m=+3785.276472221"
Oct 30 05:00:06.397706 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.397669    5072 generic.go:356] "Generic (PLEG): container finished" podID="927eaf22-dd8b-4bef-a48c-fff361ead463" containerID="11c3e6636e37e94b9b34a11c8ba06c213ec7881f2f8a9966e7afa509f06bf722" exitCode=60
Oct 30 05:00:06.398149 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.397781    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m" event={"ID":"927eaf22-dd8b-4bef-a48c-fff361ead463","Type":"ContainerDied","Data":"11c3e6636e37e94b9b34a11c8ba06c213ec7881f2f8a9966e7afa509f06bf722"}
Oct 30 05:00:06.398149 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.397869    5072 scope.go:122] "RemoveContainer" containerID="c22f88016d803962f88d3075c5a08c238d83060e2684dfd7486ff066832dcf83"
Oct 30 05:00:06.398252 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.398193    5072 scope.go:122] "RemoveContainer" containerID="11c3e6636e37e94b9b34a11c8ba06c213ec7881f2f8a9966e7afa509f06bf722"
Oct 30 05:00:06.398599 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:06.398570    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m_odh-ai-gateway-infra(927eaf22-dd8b-4bef-a48c-fff361ead463)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463"
Oct 30 05:00:06.400955 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.400933    5072 generic.go:356] "Generic (PLEG): container finished" podID="3621a9d6-ea49-415f-a6a9-5ce99245d111" containerID="591a426acee69061f04dd12a71dab7b97f9e69b1b4128d3de05b4312c743be24" exitCode=60
Oct 30 05:00:06.401349 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.401333    5072 scope.go:122] "RemoveContainer" containerID="591a426acee69061f04dd12a71dab7b97f9e69b1b4128d3de05b4312c743be24"
Oct 30 05:00:06.401711 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:06.401689    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n_odh-ai-gateway-infra(3621a9d6-ea49-415f-a6a9-5ce99245d111)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111"
Oct 30 05:00:06.403261 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.403243    5072 generic.go:356] "Generic (PLEG): container finished" podID="fc366c3b-6526-4575-899f-df191a5971de" containerID="da2e437e983141ec02fa7b21035f90ccec5f0efd176f4dd790833fabb8902c5e" exitCode=60
Oct 30 05:00:06.403592 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.403570    5072 scope.go:122] "RemoveContainer" containerID="da2e437e983141ec02fa7b21035f90ccec5f0efd176f4dd790833fabb8902c5e"
Oct 30 05:00:06.403968 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:06.403947    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2_odh-ai-gateway-infra(fc366c3b-6526-4575-899f-df191a5971de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2" podUID="fc366c3b-6526-4575-899f-df191a5971de"
Oct 30 05:00:06.405433 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.405411    5072 generic.go:356] "Generic (PLEG): container finished" podID="d8802708-e588-476f-b55b-bce1a3efa0cd" containerID="519099ca6ab822c89659ec20045ca80e7a3fcb5a0acb95987001449c13e8c014" exitCode=60
Oct 30 05:00:06.405754 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.405738    5072 scope.go:122] "RemoveContainer" containerID="519099ca6ab822c89659ec20045ca80e7a3fcb5a0acb95987001449c13e8c014"
Oct 30 05:00:06.406004 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:06.405986    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw_odh-ai-gateway-infra(d8802708-e588-476f-b55b-bce1a3efa0cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd"
Oct 30 05:00:06.407556 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.407534    5072 generic.go:356] "Generic (PLEG): container finished" podID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" containerID="9e990a2dd0ebc27338d0b1fb4c8d60135554438ebc178741dba87e83fd8bdb39" exitCode=60
Oct 30 05:00:06.407864 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.407845    5072 scope.go:122] "RemoveContainer" containerID="9e990a2dd0ebc27338d0b1fb4c8d60135554438ebc178741dba87e83fd8bdb39"
Oct 30 05:00:06.408165 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:06.408142    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk_odh-ai-gateway-infra(8dff5e9e-5890-4180-8c30-ee2b8a1b4e61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61"
Oct 30 05:00:06.409740 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.409722    5072 generic.go:356] "Generic (PLEG): container finished" podID="92e82ab9-06b8-4c71-b46a-76348dee31b5" containerID="2138d1a10a60a072b9380a25ca7eafc48ad0145135688f95fbec566c89b5a11e" exitCode=60
Oct 30 05:00:06.410037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.410025    5072 scope.go:122] "RemoveContainer" containerID="2138d1a10a60a072b9380a25ca7eafc48ad0145135688f95fbec566c89b5a11e"
Oct 30 05:00:06.410299 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:06.410282    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb_odh-ai-gateway-infra(92e82ab9-06b8-4c71-b46a-76348dee31b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5"
Oct 30 05:00:06.411783 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.411763    5072 generic.go:356] "Generic (PLEG): container finished" podID="5d6f868c-5bea-49e5-a136-ab5862d1841d" containerID="3716493d7a1d9bf72fffa52bb393512bfbcc2eb222115fd3cbfe445f2e4a0457" exitCode=60
Oct 30 05:00:06.412133 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.412070    5072 scope.go:122] "RemoveContainer" containerID="3716493d7a1d9bf72fffa52bb393512bfbcc2eb222115fd3cbfe445f2e4a0457"
Oct 30 05:00:06.412387 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:06.412366    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c_odh-ai-gateway-infra(5d6f868c-5bea-49e5-a136-ab5862d1841d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d"
Oct 30 05:00:06.413993 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.413976    5072 generic.go:356] "Generic (PLEG): container finished" podID="c9acd468-3d02-4521-9db7-33e9dff0259f" containerID="013efd90b4d50e87f984bdb52e6bd7a24878a7ed7282b6baf07dc53d30a54ad2" exitCode=60
Oct 30 05:00:06.414390 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.414371    5072 scope.go:122] "RemoveContainer" containerID="013efd90b4d50e87f984bdb52e6bd7a24878a7ed7282b6baf07dc53d30a54ad2"
Oct 30 05:00:06.414668 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:06.414632    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs_odh-ai-gateway-infra(c9acd468-3d02-4521-9db7-33e9dff0259f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs" podUID="c9acd468-3d02-4521-9db7-33e9dff0259f"
Oct 30 05:00:06.414978 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.414962    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="104bfbf8-92e2-4dab-82a7-7b2ff1771d53" path="/var/lib/kubelet/pods/104bfbf8-92e2-4dab-82a7-7b2ff1771d53/volumes"
Oct 30 05:00:06.415504 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.415481    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n" event={"ID":"3621a9d6-ea49-415f-a6a9-5ce99245d111","Type":"ContainerDied","Data":"591a426acee69061f04dd12a71dab7b97f9e69b1b4128d3de05b4312c743be24"}
Oct 30 05:00:06.415809 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.415573    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2" event={"ID":"fc366c3b-6526-4575-899f-df191a5971de","Type":"ContainerDied","Data":"da2e437e983141ec02fa7b21035f90ccec5f0efd176f4dd790833fabb8902c5e"}
Oct 30 05:00:06.415809 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.415625    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw" event={"ID":"d8802708-e588-476f-b55b-bce1a3efa0cd","Type":"ContainerDied","Data":"519099ca6ab822c89659ec20045ca80e7a3fcb5a0acb95987001449c13e8c014"}
Oct 30 05:00:06.415809 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.415656    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk" event={"ID":"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61","Type":"ContainerDied","Data":"9e990a2dd0ebc27338d0b1fb4c8d60135554438ebc178741dba87e83fd8bdb39"}
Oct 30 05:00:06.415809 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.415722    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb" event={"ID":"92e82ab9-06b8-4c71-b46a-76348dee31b5","Type":"ContainerDied","Data":"2138d1a10a60a072b9380a25ca7eafc48ad0145135688f95fbec566c89b5a11e"}
Oct 30 05:00:06.415809 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.415794    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c" event={"ID":"5d6f868c-5bea-49e5-a136-ab5862d1841d","Type":"ContainerDied","Data":"3716493d7a1d9bf72fffa52bb393512bfbcc2eb222115fd3cbfe445f2e4a0457"}
Oct 30 05:00:06.416060 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.415825    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs" event={"ID":"c9acd468-3d02-4521-9db7-33e9dff0259f","Type":"ContainerDied","Data":"013efd90b4d50e87f984bdb52e6bd7a24878a7ed7282b6baf07dc53d30a54ad2"}
Oct 30 05:00:06.416561 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.416540    5072 generic.go:356] "Generic (PLEG): container finished" podID="f638f03d-08bb-4937-aa89-2198ac4a4c02" containerID="51a84f959641959ea6068db972539d24fc2db2075adc2abda73be2f1ce19dcac" exitCode=60
Oct 30 05:00:06.416641 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.416626    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g" event={"ID":"f638f03d-08bb-4937-aa89-2198ac4a4c02","Type":"ContainerDied","Data":"51a84f959641959ea6068db972539d24fc2db2075adc2abda73be2f1ce19dcac"}
Oct 30 05:00:06.416802 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.416788    5072 scope.go:122] "RemoveContainer" containerID="51a84f959641959ea6068db972539d24fc2db2075adc2abda73be2f1ce19dcac"
Oct 30 05:00:06.417015 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:06.416991    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g_odh-ai-gateway-infra(f638f03d-08bb-4937-aa89-2198ac4a4c02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02"
Oct 30 05:00:06.418429 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.418405    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7" event={"ID":"82ca5f62-df90-42b8-91cb-dadacd5b3668","Type":"ContainerStarted","Data":"7730d03d447b4940009fcce0c0bf7d439787909417eed32da0ca3e5cc5e5aa1d"}
Oct 30 05:00:06.420136 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.420108    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h" event={"ID":"804abcc6-7570-4f94-be22-d2c8b12f2108","Type":"ContainerStarted","Data":"108e134f7eaf299304768554dcc88d15750c43815a1b9ea1780917ae70eefd03"}
Oct 30 05:00:06.422065 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.422045    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d" event={"ID":"a5cb00e5-a10c-4e5a-846b-7e10e452611e","Type":"ContainerStarted","Data":"041f5cd1a538fc86d15850849ca97c3e0a9c9d29ef4f332d6c5c3c5052ac8b53"}
Oct 30 05:00:06.423977 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.423953    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh" event={"ID":"7fdb04a5-7116-41a3-80cd-f1169b75cc2b","Type":"ContainerStarted","Data":"edc316ab392a72d45495dc6cd1e7f2646dc3bb0b55a290dfd82527c0a3e1069a"}
Oct 30 05:00:06.425853 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.425827    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt" event={"ID":"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3","Type":"ContainerStarted","Data":"3f28bad762c18973802882cd6d7ddc0cc873b5d8424ff23e799ca911093e30c0"}
Oct 30 05:00:06.427910 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.427888    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv" event={"ID":"f52b50b1-dfe7-4316-9918-61d44366cd60","Type":"ContainerStarted","Data":"39f56b4f616cb5c4e35758f5f531097cf015b3c4ef80f72fd7c903439504ab5a"}
Oct 30 05:00:06.429789 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.429764    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9" event={"ID":"212e51ce-0a69-4809-822f-6f6fe6bf8773","Type":"ContainerStarted","Data":"27475a32b37d513229f67e0d473cf5a4f1d45659579df7ace12a3212c32a8d56"}
Oct 30 05:00:06.432891 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.432868    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" event={"ID":"0e846f7c-186a-4e49-ba15-6ff6c564bc13","Type":"ContainerStarted","Data":"b158b31edd6d6339c1588538cf92618a47b63783c2011f73cd72054dbaf5a737"}
Oct 30 05:00:06.434640 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.434619    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h" event={"ID":"06e8c6d6-1834-4ac0-a21d-998da69a4379","Type":"ContainerStarted","Data":"578a7498bedac1175fd73411eddb28b00095e97be0a2d1b39201bb8d258d1294"}
Oct 30 05:00:06.436590 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.436568    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf" event={"ID":"2a82ff41-ac42-48bc-81c4-24e539017282","Type":"ContainerStarted","Data":"20d2ed910c168a11ba67f791a63dd0983d7a684d40f27cc9774cf9f5670d7040"}
Oct 30 05:00:06.438472 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.438450    5072 generic.go:356] "Generic (PLEG): container finished" podID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" containerID="16e10d0b9c9bd56da65bf95af053716430b0dbf497d5bc33241e767f8b09bf49" exitCode=60
Oct 30 05:00:06.438552 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.438480    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9" event={"ID":"6fa98861-f2c2-47aa-bb83-54ddf8c5c715","Type":"ContainerDied","Data":"16e10d0b9c9bd56da65bf95af053716430b0dbf497d5bc33241e767f8b09bf49"}
Oct 30 05:00:06.438712 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.438694    5072 scope.go:122] "RemoveContainer" containerID="16e10d0b9c9bd56da65bf95af053716430b0dbf497d5bc33241e767f8b09bf49"
Oct 30 05:00:06.438917 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:06.438900    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9_odh-ai-gateway-infra(6fa98861-f2c2-47aa-bb83-54ddf8c5c715)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9" podUID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715"
Oct 30 05:00:06.440616 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.440594    5072 generic.go:356] "Generic (PLEG): container finished" podID="e53d6987-db5e-468a-9afe-2c0603cbf408" containerID="8b48e76ffc3256ff64f90aaf9f650b41e6f353ebebc176b861874f746cb85f2b" exitCode=60
Oct 30 05:00:06.440699 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.440659    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5" event={"ID":"e53d6987-db5e-468a-9afe-2c0603cbf408","Type":"ContainerDied","Data":"8b48e76ffc3256ff64f90aaf9f650b41e6f353ebebc176b861874f746cb85f2b"}
Oct 30 05:00:06.440914 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.440895    5072 scope.go:122] "RemoveContainer" containerID="8b48e76ffc3256ff64f90aaf9f650b41e6f353ebebc176b861874f746cb85f2b"
Oct 30 05:00:06.441195 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:06.441170    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5_odh-ai-gateway-infra(e53d6987-db5e-468a-9afe-2c0603cbf408)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408"
Oct 30 05:00:06.443128 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.443105    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw" event={"ID":"7a540e59-8239-423a-ac80-aa84d7b4821d","Type":"ContainerDied","Data":"d365486c30671c65b714592a62e9ed7113fb64bddae0406abafdd0568eb4fbd5"}
Oct 30 05:00:06.443220 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.443071    5072 generic.go:356] "Generic (PLEG): container finished" podID="7a540e59-8239-423a-ac80-aa84d7b4821d" containerID="d365486c30671c65b714592a62e9ed7113fb64bddae0406abafdd0568eb4fbd5" exitCode=60
Oct 30 05:00:06.443370 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.443353    5072 scope.go:122] "RemoveContainer" containerID="d365486c30671c65b714592a62e9ed7113fb64bddae0406abafdd0568eb4fbd5"
Oct 30 05:00:06.443580 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:06.443560    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw_odh-ai-gateway-infra(7a540e59-8239-423a-ac80-aa84d7b4821d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d"
Oct 30 05:00:06.454914 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.454893    5072 scope.go:122] "RemoveContainer" containerID="138210d3db7aca72a22226d1504ba5759eee3e6abc5dd8a3f3b79281dc8b57ec"
Oct 30 05:00:06.610267 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.610172    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf" podStartSLOduration=6.610158437 podStartE2EDuration="6.610158437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:06.6086037 +0000 UTC m=+3786.350026797" watchObservedRunningTime="2026-10-30 05:00:06.610158437 +0000 UTC m=+3786.351581534"
Oct 30 05:00:06.653740 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.653679    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h" podStartSLOduration=6.653665312 podStartE2EDuration="6.653665312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:06.650376778 +0000 UTC m=+3786.391799890" watchObservedRunningTime="2026-10-30 05:00:06.653665312 +0000 UTC m=+3786.395088410"
Oct 30 05:00:06.947454 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.947406    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh" podStartSLOduration=6.947391154 podStartE2EDuration="6.947391154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:06.944178575 +0000 UTC m=+3786.685601672" watchObservedRunningTime="2026-10-30 05:00:06.947391154 +0000 UTC m=+3786.688814250"
Oct 30 05:00:06.989354 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:06.989303    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9" podStartSLOduration=6.989289701 podStartE2EDuration="6.989289701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:06.988308269 +0000 UTC m=+3786.729731365" watchObservedRunningTime="2026-10-30 05:00:06.989289701 +0000 UTC m=+3786.730712798"
Oct 30 05:00:07.007409 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.007280    5072 scope.go:122] "RemoveContainer" containerID="6f52ec0b06275a65c38aa66fa8e4430e3c4c538cf49e9a299b4f1eeb73b9160a"
Oct 30 05:00:07.023797 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.023775    5072 scope.go:122] "RemoveContainer" containerID="d3f642956f0df5de2f96de63ba433193c832d0489c44ee490623b112e9f89a9f"
Oct 30 05:00:07.039866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.039844    5072 scope.go:122] "RemoveContainer" containerID="9930286fdd71012fa66ad8a32e39a8d6cbf8beaabdb66b2fbee6ac30e773ea20"
Oct 30 05:00:07.055548 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.055527    5072 scope.go:122] "RemoveContainer" containerID="392e05a495e6d8bb7bc2ddcd802e20e7f1c80ed65220b590b172ee786faa5eb1"
Oct 30 05:00:07.071542 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.071520    5072 scope.go:122] "RemoveContainer" containerID="e042f4d711102e125592724cbbb8dedbcb0d2ef58a7294e6cefe939a6edc103c"
Oct 30 05:00:07.089219 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.089200    5072 scope.go:122] "RemoveContainer" containerID="608924cc655a3d257c4bf7d6dbbbfb97a51553b28a9e2cbf9cd55dcc7f3c9429"
Oct 30 05:00:07.105575 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.105550    5072 scope.go:122] "RemoveContainer" containerID="f841a7f5b7bb8cc5b63475258a56e29340904242854275b6c8ddf25ba98d99e7"
Oct 30 05:00:07.120253 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.120232    5072 scope.go:122] "RemoveContainer" containerID="602ad394c87089f452b930f3b94b0c108fd515484d260a700ff8f254dc947814"
Oct 30 05:00:07.127347 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.127305    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7" podStartSLOduration=7.127292383 podStartE2EDuration="7.127292383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:07.124368914 +0000 UTC m=+3786.865792012" watchObservedRunningTime="2026-10-30 05:00:07.127292383 +0000 UTC m=+3786.868715490"
Oct 30 05:00:07.134916 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.134894    5072 scope.go:122] "RemoveContainer" containerID="950263a077853fffb7d0f94d647544951e930c7bf204aff3a8d8caed3c4ca95d"
Oct 30 05:00:07.150145 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.150122    5072 scope.go:122] "RemoveContainer" containerID="68e4bfd767bd9c8e62cfd6f87426c64ce8dd18b4274a60d76077afeccdc79ee8"
Oct 30 05:00:07.460418 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.460383    5072 scope.go:122] "RemoveContainer" containerID="8b48e76ffc3256ff64f90aaf9f650b41e6f353ebebc176b861874f746cb85f2b"
Oct 30 05:00:07.460896 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.460688    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5_odh-ai-gateway-infra(e53d6987-db5e-468a-9afe-2c0603cbf408)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408"
Oct 30 05:00:07.462128 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.462108    5072 scope.go:122] "RemoveContainer" containerID="d365486c30671c65b714592a62e9ed7113fb64bddae0406abafdd0568eb4fbd5"
Oct 30 05:00:07.462382 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.462364    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw_odh-ai-gateway-infra(7a540e59-8239-423a-ac80-aa84d7b4821d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d"
Oct 30 05:00:07.463828 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.463805    5072 scope.go:122] "RemoveContainer" containerID="11c3e6636e37e94b9b34a11c8ba06c213ec7881f2f8a9966e7afa509f06bf722"
Oct 30 05:00:07.464082 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.464058    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m_odh-ai-gateway-infra(927eaf22-dd8b-4bef-a48c-fff361ead463)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463"
Oct 30 05:00:07.465440 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.465423    5072 scope.go:122] "RemoveContainer" containerID="591a426acee69061f04dd12a71dab7b97f9e69b1b4128d3de05b4312c743be24"
Oct 30 05:00:07.465673 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.465650    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n_odh-ai-gateway-infra(3621a9d6-ea49-415f-a6a9-5ce99245d111)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111"
Oct 30 05:00:07.466968 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.466953    5072 scope.go:122] "RemoveContainer" containerID="da2e437e983141ec02fa7b21035f90ccec5f0efd176f4dd790833fabb8902c5e"
Oct 30 05:00:07.467226 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.467204    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2_odh-ai-gateway-infra(fc366c3b-6526-4575-899f-df191a5971de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2" podUID="fc366c3b-6526-4575-899f-df191a5971de"
Oct 30 05:00:07.468551 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.468534    5072 scope.go:122] "RemoveContainer" containerID="519099ca6ab822c89659ec20045ca80e7a3fcb5a0acb95987001449c13e8c014"
Oct 30 05:00:07.468757 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.468737    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw_odh-ai-gateway-infra(d8802708-e588-476f-b55b-bce1a3efa0cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd"
Oct 30 05:00:07.470139 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.470121    5072 scope.go:122] "RemoveContainer" containerID="9e990a2dd0ebc27338d0b1fb4c8d60135554438ebc178741dba87e83fd8bdb39"
Oct 30 05:00:07.470375 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.470354    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk_odh-ai-gateway-infra(8dff5e9e-5890-4180-8c30-ee2b8a1b4e61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61"
Oct 30 05:00:07.471847 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.471828    5072 scope.go:122] "RemoveContainer" containerID="2138d1a10a60a072b9380a25ca7eafc48ad0145135688f95fbec566c89b5a11e"
Oct 30 05:00:07.472135 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.472110    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb_odh-ai-gateway-infra(92e82ab9-06b8-4c71-b46a-76348dee31b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5"
Oct 30 05:00:07.473536 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.473517    5072 scope.go:122] "RemoveContainer" containerID="3716493d7a1d9bf72fffa52bb393512bfbcc2eb222115fd3cbfe445f2e4a0457"
Oct 30 05:00:07.473738 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.473715    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c_odh-ai-gateway-infra(5d6f868c-5bea-49e5-a136-ab5862d1841d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d"
Oct 30 05:00:07.475183 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.475166    5072 scope.go:122] "RemoveContainer" containerID="013efd90b4d50e87f984bdb52e6bd7a24878a7ed7282b6baf07dc53d30a54ad2"
Oct 30 05:00:07.475407 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.475386    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs_odh-ai-gateway-infra(c9acd468-3d02-4521-9db7-33e9dff0259f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs" podUID="c9acd468-3d02-4521-9db7-33e9dff0259f"
Oct 30 05:00:07.476714 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.476697    5072 scope.go:122] "RemoveContainer" containerID="51a84f959641959ea6068db972539d24fc2db2075adc2abda73be2f1ce19dcac"
Oct 30 05:00:07.476922 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.476902    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g_odh-ai-gateway-infra(f638f03d-08bb-4937-aa89-2198ac4a4c02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02"
Oct 30 05:00:07.478189 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.478166    5072 generic.go:356] "Generic (PLEG): container finished" podID="82ca5f62-df90-42b8-91cb-dadacd5b3668" containerID="7730d03d447b4940009fcce0c0bf7d439787909417eed32da0ca3e5cc5e5aa1d" exitCode=60
Oct 30 05:00:07.478280 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.478251    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7" event={"ID":"82ca5f62-df90-42b8-91cb-dadacd5b3668","Type":"ContainerDied","Data":"7730d03d447b4940009fcce0c0bf7d439787909417eed32da0ca3e5cc5e5aa1d"}
Oct 30 05:00:07.478338 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.478308    5072 scope.go:122] "RemoveContainer" containerID="9cc782b6bca432cf512cba6ffcbccb9c6481bab8baf9b3c77a729b98196e2abd"
Oct 30 05:00:07.478447 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.478426    5072 scope.go:122] "RemoveContainer" containerID="7730d03d447b4940009fcce0c0bf7d439787909417eed32da0ca3e5cc5e5aa1d"
Oct 30 05:00:07.478710 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.478685    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7_odh-ai-gateway-infra(82ca5f62-df90-42b8-91cb-dadacd5b3668)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668"
Oct 30 05:00:07.480025 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.480006    5072 generic.go:356] "Generic (PLEG): container finished" podID="804abcc6-7570-4f94-be22-d2c8b12f2108" containerID="108e134f7eaf299304768554dcc88d15750c43815a1b9ea1780917ae70eefd03" exitCode=60
Oct 30 05:00:07.480119 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.480069    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h" event={"ID":"804abcc6-7570-4f94-be22-d2c8b12f2108","Type":"ContainerDied","Data":"108e134f7eaf299304768554dcc88d15750c43815a1b9ea1780917ae70eefd03"}
Oct 30 05:00:07.480266 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.480253    5072 scope.go:122] "RemoveContainer" containerID="108e134f7eaf299304768554dcc88d15750c43815a1b9ea1780917ae70eefd03"
Oct 30 05:00:07.480491 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.480471    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h_odh-ai-gateway-infra(804abcc6-7570-4f94-be22-d2c8b12f2108)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108"
Oct 30 05:00:07.481867 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.481846    5072 generic.go:356] "Generic (PLEG): container finished" podID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" containerID="041f5cd1a538fc86d15850849ca97c3e0a9c9d29ef4f332d6c5c3c5052ac8b53" exitCode=60
Oct 30 05:00:07.481959 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.481920    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d" event={"ID":"a5cb00e5-a10c-4e5a-846b-7e10e452611e","Type":"ContainerDied","Data":"041f5cd1a538fc86d15850849ca97c3e0a9c9d29ef4f332d6c5c3c5052ac8b53"}
Oct 30 05:00:07.482147 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.482132    5072 scope.go:122] "RemoveContainer" containerID="041f5cd1a538fc86d15850849ca97c3e0a9c9d29ef4f332d6c5c3c5052ac8b53"
Oct 30 05:00:07.482352 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.482332    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d_odh-ai-gateway-infra(a5cb00e5-a10c-4e5a-846b-7e10e452611e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e"
Oct 30 05:00:07.483867 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.483844    5072 generic.go:356] "Generic (PLEG): container finished" podID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" containerID="edc316ab392a72d45495dc6cd1e7f2646dc3bb0b55a290dfd82527c0a3e1069a" exitCode=60
Oct 30 05:00:07.483953 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.483880    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh" event={"ID":"7fdb04a5-7116-41a3-80cd-f1169b75cc2b","Type":"ContainerDied","Data":"edc316ab392a72d45495dc6cd1e7f2646dc3bb0b55a290dfd82527c0a3e1069a"}
Oct 30 05:00:07.484123 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.484109    5072 scope.go:122] "RemoveContainer" containerID="edc316ab392a72d45495dc6cd1e7f2646dc3bb0b55a290dfd82527c0a3e1069a"
Oct 30 05:00:07.484360 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.484338    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888940-bbnrh_odh-ai-gateway-infra(7fdb04a5-7116-41a3-80cd-f1169b75cc2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b"
Oct 30 05:00:07.485988 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.485965    5072 generic.go:356] "Generic (PLEG): container finished" podID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" containerID="3f28bad762c18973802882cd6d7ddc0cc873b5d8424ff23e799ca911093e30c0" exitCode=60
Oct 30 05:00:07.486109 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.486015    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt" event={"ID":"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3","Type":"ContainerDied","Data":"3f28bad762c18973802882cd6d7ddc0cc873b5d8424ff23e799ca911093e30c0"}
Oct 30 05:00:07.486399 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.486382    5072 scope.go:122] "RemoveContainer" containerID="3f28bad762c18973802882cd6d7ddc0cc873b5d8424ff23e799ca911093e30c0"
Oct 30 05:00:07.486643 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.486624    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt_odh-ai-gateway-infra(ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3"
Oct 30 05:00:07.488191 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.488165    5072 generic.go:356] "Generic (PLEG): container finished" podID="f52b50b1-dfe7-4316-9918-61d44366cd60" containerID="39f56b4f616cb5c4e35758f5f531097cf015b3c4ef80f72fd7c903439504ab5a" exitCode=60
Oct 30 05:00:07.488351 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.488247    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv" event={"ID":"f52b50b1-dfe7-4316-9918-61d44366cd60","Type":"ContainerDied","Data":"39f56b4f616cb5c4e35758f5f531097cf015b3c4ef80f72fd7c903439504ab5a"}
Oct 30 05:00:07.488466 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.488446    5072 scope.go:122] "RemoveContainer" containerID="39f56b4f616cb5c4e35758f5f531097cf015b3c4ef80f72fd7c903439504ab5a"
Oct 30 05:00:07.488698 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.488679    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv_odh-ai-gateway-infra(f52b50b1-dfe7-4316-9918-61d44366cd60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60"
Oct 30 05:00:07.490636 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.490617    5072 generic.go:356] "Generic (PLEG): container finished" podID="212e51ce-0a69-4809-822f-6f6fe6bf8773" containerID="27475a32b37d513229f67e0d473cf5a4f1d45659579df7ace12a3212c32a8d56" exitCode=60
Oct 30 05:00:07.490727 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.490688    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9" event={"ID":"212e51ce-0a69-4809-822f-6f6fe6bf8773","Type":"ContainerDied","Data":"27475a32b37d513229f67e0d473cf5a4f1d45659579df7ace12a3212c32a8d56"}
Oct 30 05:00:07.490896 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.490879    5072 scope.go:122] "RemoveContainer" containerID="27475a32b37d513229f67e0d473cf5a4f1d45659579df7ace12a3212c32a8d56"
Oct 30 05:00:07.491133 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.491111    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9_odh-ai-gateway-infra(212e51ce-0a69-4809-822f-6f6fe6bf8773)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9" podUID="212e51ce-0a69-4809-822f-6f6fe6bf8773"
Oct 30 05:00:07.492782 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.492763    5072 generic.go:356] "Generic (PLEG): container finished" podID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" containerID="b158b31edd6d6339c1588538cf92618a47b63783c2011f73cd72054dbaf5a737" exitCode=60
Oct 30 05:00:07.492866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.492830    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" event={"ID":"0e846f7c-186a-4e49-ba15-6ff6c564bc13","Type":"ContainerDied","Data":"b158b31edd6d6339c1588538cf92618a47b63783c2011f73cd72054dbaf5a737"}
Oct 30 05:00:07.493101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.493066    5072 scope.go:122] "RemoveContainer" containerID="b158b31edd6d6339c1588538cf92618a47b63783c2011f73cd72054dbaf5a737"
Oct 30 05:00:07.493350 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.493329    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6_odh-ai-gateway-infra(0e846f7c-186a-4e49-ba15-6ff6c564bc13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13"
Oct 30 05:00:07.494968 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.494946    5072 generic.go:356] "Generic (PLEG): container finished" podID="06e8c6d6-1834-4ac0-a21d-998da69a4379" containerID="578a7498bedac1175fd73411eddb28b00095e97be0a2d1b39201bb8d258d1294" exitCode=60
Oct 30 05:00:07.495070 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.495004    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h" event={"ID":"06e8c6d6-1834-4ac0-a21d-998da69a4379","Type":"ContainerDied","Data":"578a7498bedac1175fd73411eddb28b00095e97be0a2d1b39201bb8d258d1294"}
Oct 30 05:00:07.495284 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.495266    5072 scope.go:122] "RemoveContainer" containerID="578a7498bedac1175fd73411eddb28b00095e97be0a2d1b39201bb8d258d1294"
Oct 30 05:00:07.495567 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.495538    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h_odh-ai-gateway-infra(06e8c6d6-1834-4ac0-a21d-998da69a4379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379"
Oct 30 05:00:07.496509 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.496495    5072 scope.go:122] "RemoveContainer" containerID="194a5e992ac1667fbfc3318ee18418fda85740a6e129f772d7b1f95fa7b2848f"
Oct 30 05:00:07.497017 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.496997    5072 generic.go:356] "Generic (PLEG): container finished" podID="2a82ff41-ac42-48bc-81c4-24e539017282" containerID="20d2ed910c168a11ba67f791a63dd0983d7a684d40f27cc9774cf9f5670d7040" exitCode=60
Oct 30 05:00:07.497128 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.497105    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf" event={"ID":"2a82ff41-ac42-48bc-81c4-24e539017282","Type":"ContainerDied","Data":"20d2ed910c168a11ba67f791a63dd0983d7a684d40f27cc9774cf9f5670d7040"}
Oct 30 05:00:07.497299 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.497277    5072 scope.go:122] "RemoveContainer" containerID="20d2ed910c168a11ba67f791a63dd0983d7a684d40f27cc9774cf9f5670d7040"
Oct 30 05:00:07.497479 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.497458    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf_odh-ai-gateway-infra(2a82ff41-ac42-48bc-81c4-24e539017282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf" podUID="2a82ff41-ac42-48bc-81c4-24e539017282"
Oct 30 05:00:07.498935 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.498917    5072 scope.go:122] "RemoveContainer" containerID="16e10d0b9c9bd56da65bf95af053716430b0dbf497d5bc33241e767f8b09bf49"
Oct 30 05:00:07.499163 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:07.499143    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9_odh-ai-gateway-infra(6fa98861-f2c2-47aa-bb83-54ddf8c5c715)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9" podUID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715"
Oct 30 05:00:07.512678 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.512659    5072 scope.go:122] "RemoveContainer" containerID="757a59225b257b8a0696291740cb104a3cf9da536d54f0996ee2cd7b30b2e402"
Oct 30 05:00:07.528801 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.528781    5072 scope.go:122] "RemoveContainer" containerID="dace9df966feb7f31457c1c803304e48fd5dc4b550944dd5f51d185dd5bf5186"
Oct 30 05:00:07.544686 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.544666    5072 scope.go:122] "RemoveContainer" containerID="9dbaa42c4382ff1c0d6fe092a4d0a271faa9b0a66fd741afe36b939a9162fac0"
Oct 30 05:00:07.559600 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.559572    5072 scope.go:122] "RemoveContainer" containerID="e669a446601af077799f9626ed1d05d829bc8730f72a6ba65926c5daa0310cc6"
Oct 30 05:00:07.575358 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.575339    5072 scope.go:122] "RemoveContainer" containerID="986a374e1101aca1ba3000fa3f5748453a2a561a108c440ea93937004ddbc2ef"
Oct 30 05:00:07.589620 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.589589    5072 scope.go:122] "RemoveContainer" containerID="a6f45048bbf003aff2dae5aeff52f7fe751feac63ade2dbeb1ff74757b8c1bbf"
Oct 30 05:00:07.605054 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.605030    5072 scope.go:122] "RemoveContainer" containerID="4528a89ba0b1bebd3cda0fdaf74c08f14bee541ede5f7a90d7283516917b001c"
Oct 30 05:00:07.622102 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:07.622046    5072 scope.go:122] "RemoveContainer" containerID="f2ac90afd86c848d307a7a7c1a6d2716960c4802d3a44dd66ab09ee548c3da7d"
Oct 30 05:00:08.510357 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:08.510325    5072 scope.go:122] "RemoveContainer" containerID="7730d03d447b4940009fcce0c0bf7d439787909417eed32da0ca3e5cc5e5aa1d"
Oct 30 05:00:08.510849 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:08.510713    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7_odh-ai-gateway-infra(82ca5f62-df90-42b8-91cb-dadacd5b3668)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668"
Oct 30 05:00:08.512390 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:08.512362    5072 scope.go:122] "RemoveContainer" containerID="108e134f7eaf299304768554dcc88d15750c43815a1b9ea1780917ae70eefd03"
Oct 30 05:00:08.512764 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:08.512737    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h_odh-ai-gateway-infra(804abcc6-7570-4f94-be22-d2c8b12f2108)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108"
Oct 30 05:00:08.514964 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:08.514939    5072 scope.go:122] "RemoveContainer" containerID="041f5cd1a538fc86d15850849ca97c3e0a9c9d29ef4f332d6c5c3c5052ac8b53"
Oct 30 05:00:08.515316 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:08.515290    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d_odh-ai-gateway-infra(a5cb00e5-a10c-4e5a-846b-7e10e452611e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e"
Oct 30 05:00:08.516877 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:08.516856    5072 scope.go:122] "RemoveContainer" containerID="edc316ab392a72d45495dc6cd1e7f2646dc3bb0b55a290dfd82527c0a3e1069a"
Oct 30 05:00:08.517174 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:08.517139    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888940-bbnrh_odh-ai-gateway-infra(7fdb04a5-7116-41a3-80cd-f1169b75cc2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b"
Oct 30 05:00:08.518682 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:08.518660    5072 scope.go:122] "RemoveContainer" containerID="3f28bad762c18973802882cd6d7ddc0cc873b5d8424ff23e799ca911093e30c0"
Oct 30 05:00:08.518939 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:08.518916    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt_odh-ai-gateway-infra(ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3"
Oct 30 05:00:08.520421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:08.520395    5072 scope.go:122] "RemoveContainer" containerID="39f56b4f616cb5c4e35758f5f531097cf015b3c4ef80f72fd7c903439504ab5a"
Oct 30 05:00:08.520647 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:08.520629    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv_odh-ai-gateway-infra(f52b50b1-dfe7-4316-9918-61d44366cd60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60"
Oct 30 05:00:08.522428 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:08.522409    5072 scope.go:122] "RemoveContainer" containerID="27475a32b37d513229f67e0d473cf5a4f1d45659579df7ace12a3212c32a8d56"
Oct 30 05:00:08.522692 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:08.522670    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9_odh-ai-gateway-infra(212e51ce-0a69-4809-822f-6f6fe6bf8773)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9" podUID="212e51ce-0a69-4809-822f-6f6fe6bf8773"
Oct 30 05:00:08.524207 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:08.524164    5072 scope.go:122] "RemoveContainer" containerID="b158b31edd6d6339c1588538cf92618a47b63783c2011f73cd72054dbaf5a737"
Oct 30 05:00:08.524441 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:08.524413    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6_odh-ai-gateway-infra(0e846f7c-186a-4e49-ba15-6ff6c564bc13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13"
Oct 30 05:00:08.525854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:08.525836    5072 scope.go:122] "RemoveContainer" containerID="578a7498bedac1175fd73411eddb28b00095e97be0a2d1b39201bb8d258d1294"
Oct 30 05:00:08.526109 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:08.526056    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h_odh-ai-gateway-infra(06e8c6d6-1834-4ac0-a21d-998da69a4379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379"
Oct 30 05:00:08.527489 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:08.527473    5072 scope.go:122] "RemoveContainer" containerID="20d2ed910c168a11ba67f791a63dd0983d7a684d40f27cc9774cf9f5670d7040"
Oct 30 05:00:08.527678 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:08.527660    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf_odh-ai-gateway-infra(2a82ff41-ac42-48bc-81c4-24e539017282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf" podUID="2a82ff41-ac42-48bc-81c4-24e539017282"
Oct 30 05:00:11.753333 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:11.753295    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:00:16.413333 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:16.413305    5072 scope.go:122] "RemoveContainer" containerID="519099ca6ab822c89659ec20045ca80e7a3fcb5a0acb95987001449c13e8c014"
Oct 30 05:00:16.413766 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:16.413369    5072 scope.go:122] "RemoveContainer" containerID="3716493d7a1d9bf72fffa52bb393512bfbcc2eb222115fd3cbfe445f2e4a0457"
Oct 30 05:00:17.401492 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:17.401454    5072 scope.go:122] "RemoveContainer" containerID="108e134f7eaf299304768554dcc88d15750c43815a1b9ea1780917ae70eefd03"
Oct 30 05:00:17.401774 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:17.401758    5072 scope.go:122] "RemoveContainer" containerID="51a84f959641959ea6068db972539d24fc2db2075adc2abda73be2f1ce19dcac"
Oct 30 05:00:17.401823 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:17.401809    5072 scope.go:122] "RemoveContainer" containerID="d365486c30671c65b714592a62e9ed7113fb64bddae0406abafdd0568eb4fbd5"
Oct 30 05:00:17.401880 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:17.401860    5072 scope.go:122] "RemoveContainer" containerID="2138d1a10a60a072b9380a25ca7eafc48ad0145135688f95fbec566c89b5a11e"
Oct 30 05:00:17.616796 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:17.616719    5072 generic.go:356] "Generic (PLEG): container finished" podID="d8802708-e588-476f-b55b-bce1a3efa0cd" containerID="5cefd7387af4cc2d173da30cf9a5698eaba7de6ae0255561232cc8a2e50c53b3" exitCode=60
Oct 30 05:00:17.617279 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:17.616787    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw" event={"ID":"d8802708-e588-476f-b55b-bce1a3efa0cd","Type":"ContainerDied","Data":"5cefd7387af4cc2d173da30cf9a5698eaba7de6ae0255561232cc8a2e50c53b3"}
Oct 30 05:00:17.617279 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:17.616849    5072 scope.go:122] "RemoveContainer" containerID="519099ca6ab822c89659ec20045ca80e7a3fcb5a0acb95987001449c13e8c014"
Oct 30 05:00:17.617464 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:17.617362    5072 scope.go:122] "RemoveContainer" containerID="5cefd7387af4cc2d173da30cf9a5698eaba7de6ae0255561232cc8a2e50c53b3"
Oct 30 05:00:17.617821 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:17.617759    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw_odh-ai-gateway-infra(d8802708-e588-476f-b55b-bce1a3efa0cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd"
Oct 30 05:00:17.619399 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:17.619364    5072 generic.go:356] "Generic (PLEG): container finished" podID="5d6f868c-5bea-49e5-a136-ab5862d1841d" containerID="630388f646d3171193447aa2c61672d7ceed40606a008d6602a97a8d3f52864f" exitCode=60
Oct 30 05:00:17.619490 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:17.619434    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c" event={"ID":"5d6f868c-5bea-49e5-a136-ab5862d1841d","Type":"ContainerDied","Data":"630388f646d3171193447aa2c61672d7ceed40606a008d6602a97a8d3f52864f"}
Oct 30 05:00:17.619846 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:17.619830    5072 scope.go:122] "RemoveContainer" containerID="630388f646d3171193447aa2c61672d7ceed40606a008d6602a97a8d3f52864f"
Oct 30 05:00:17.620196 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:17.620179    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c_odh-ai-gateway-infra(5d6f868c-5bea-49e5-a136-ab5862d1841d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d"
Oct 30 05:00:17.734691 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:17.734669    5072 scope.go:122] "RemoveContainer" containerID="3716493d7a1d9bf72fffa52bb393512bfbcc2eb222115fd3cbfe445f2e4a0457"
Oct 30 05:00:18.413234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.413153    5072 scope.go:122] "RemoveContainer" containerID="edc316ab392a72d45495dc6cd1e7f2646dc3bb0b55a290dfd82527c0a3e1069a"
Oct 30 05:00:18.413234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.413222    5072 scope.go:122] "RemoveContainer" containerID="578a7498bedac1175fd73411eddb28b00095e97be0a2d1b39201bb8d258d1294"
Oct 30 05:00:18.413456 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.413304    5072 scope.go:122] "RemoveContainer" containerID="9e990a2dd0ebc27338d0b1fb4c8d60135554438ebc178741dba87e83fd8bdb39"
Oct 30 05:00:18.413456 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.413382    5072 scope.go:122] "RemoveContainer" containerID="b158b31edd6d6339c1588538cf92618a47b63783c2011f73cd72054dbaf5a737"
Oct 30 05:00:18.631722 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.631684    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h" event={"ID":"804abcc6-7570-4f94-be22-d2c8b12f2108","Type":"ContainerStarted","Data":"48a156daf2fd4b9964073d5e870a1170237ef4322aa18a5514a6adb1f1894641"}
Oct 30 05:00:18.633968 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.633937    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw" event={"ID":"7a540e59-8239-423a-ac80-aa84d7b4821d","Type":"ContainerStarted","Data":"15b7d378087542d122c41de65b56ca140b383b0d4475373db8c61be19f4f1f15"}
Oct 30 05:00:18.637936 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.637912    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb" event={"ID":"92e82ab9-06b8-4c71-b46a-76348dee31b5","Type":"ContainerStarted","Data":"e90064d5e85ca2d22580eb16e5a6e8312d31a3d176dc65995b49ebb1d3faea41"}
Oct 30 05:00:18.641545 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.641523    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g" event={"ID":"f638f03d-08bb-4937-aa89-2198ac4a4c02","Type":"ContainerStarted","Data":"eca3494836a00696569f03a166adc2052d62c6fb21f856b5342c0b14fd9cd3fa"}
Oct 30 05:00:18.673139 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.673008    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"]
Oct 30 05:00:18.714471 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.714435    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"]
Oct 30 05:00:18.721745 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.721694    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g" podStartSLOduration=18.72167935 podStartE2EDuration="18.72167935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:18.716677182 +0000 UTC m=+3798.458100280" watchObservedRunningTime="2026-10-30 05:00:18.72167935 +0000 UTC m=+3798.463102459"
Oct 30 05:00:18.982867 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.982840    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"
Oct 30 05:00:18.987165 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:18.987141    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"
Oct 30 05:00:19.132625 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.132597    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8802708-e588-476f-b55b-bce1a3efa0cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8802708-e588-476f-b55b-bce1a3efa0cd-maas-api-service-ca\") pod \"d8802708-e588-476f-b55b-bce1a3efa0cd\" (UID: \"d8802708-e588-476f-b55b-bce1a3efa0cd\") "
Oct 30 05:00:19.132820 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.132791    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6f868c-5bea-49e5-a136-ab5862d1841d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6f868c-5bea-49e5-a136-ab5862d1841d-maas-api-service-ca\") pod \"5d6f868c-5bea-49e5-a136-ab5862d1841d\" (UID: \"5d6f868c-5bea-49e5-a136-ab5862d1841d\") "
Oct 30 05:00:19.132895 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.132835    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6f868c-5bea-49e5-a136-ab5862d1841d-kube-api-access-pmn4h\" (UniqueName: \"kubernetes.io/projected/5d6f868c-5bea-49e5-a136-ab5862d1841d-kube-api-access-pmn4h\") pod \"5d6f868c-5bea-49e5-a136-ab5862d1841d\" (UID: \"5d6f868c-5bea-49e5-a136-ab5862d1841d\") "
Oct 30 05:00:19.132950 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.132933    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8802708-e588-476f-b55b-bce1a3efa0cd-kube-api-access-m257r\" (UniqueName: \"kubernetes.io/projected/d8802708-e588-476f-b55b-bce1a3efa0cd-kube-api-access-m257r\") pod \"d8802708-e588-476f-b55b-bce1a3efa0cd\" (UID: \"d8802708-e588-476f-b55b-bce1a3efa0cd\") "
Oct 30 05:00:19.133023 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.132932    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8802708-e588-476f-b55b-bce1a3efa0cd-maas-api-service-ca" pod "d8802708-e588-476f-b55b-bce1a3efa0cd" (UID: "d8802708-e588-476f-b55b-bce1a3efa0cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:19.133258 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.133235    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6f868c-5bea-49e5-a136-ab5862d1841d-maas-api-service-ca" pod "5d6f868c-5bea-49e5-a136-ab5862d1841d" (UID: "5d6f868c-5bea-49e5-a136-ab5862d1841d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:19.133574 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.133557    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8802708-e588-476f-b55b-bce1a3efa0cd-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:19.133639 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.133576    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6f868c-5bea-49e5-a136-ab5862d1841d-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:19.135201 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.135181    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8802708-e588-476f-b55b-bce1a3efa0cd-kube-api-access-m257r" pod "d8802708-e588-476f-b55b-bce1a3efa0cd" (UID: "d8802708-e588-476f-b55b-bce1a3efa0cd"). InnerVolumeSpecName "kube-api-access-m257r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:19.135254 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.135213    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6f868c-5bea-49e5-a136-ab5862d1841d-kube-api-access-pmn4h" pod "5d6f868c-5bea-49e5-a136-ab5862d1841d" (UID: "5d6f868c-5bea-49e5-a136-ab5862d1841d"). InnerVolumeSpecName "kube-api-access-pmn4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:19.234600 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.234571    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmn4h\" (UniqueName: \"kubernetes.io/projected/5d6f868c-5bea-49e5-a136-ab5862d1841d-kube-api-access-pmn4h\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:19.234600 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.234598    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m257r\" (UniqueName: \"kubernetes.io/projected/d8802708-e588-476f-b55b-bce1a3efa0cd-kube-api-access-m257r\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:19.400589 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.400557    5072 scope.go:122] "RemoveContainer" containerID="16e10d0b9c9bd56da65bf95af053716430b0dbf497d5bc33241e767f8b09bf49"
Oct 30 05:00:19.400745 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.400710    5072 scope.go:122] "RemoveContainer" containerID="da2e437e983141ec02fa7b21035f90ccec5f0efd176f4dd790833fabb8902c5e"
Oct 30 05:00:19.400913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.400894    5072 scope.go:122] "RemoveContainer" containerID="27475a32b37d513229f67e0d473cf5a4f1d45659579df7ace12a3212c32a8d56"
Oct 30 05:00:19.401050 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.401033    5072 scope.go:122] "RemoveContainer" containerID="3f28bad762c18973802882cd6d7ddc0cc873b5d8424ff23e799ca911093e30c0"
Oct 30 05:00:19.401206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.401180    5072 scope.go:122] "RemoveContainer" containerID="591a426acee69061f04dd12a71dab7b97f9e69b1b4128d3de05b4312c743be24"
Oct 30 05:00:19.652567 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.652536    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"
Oct 30 05:00:19.652567 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.652548    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw" event={"ID":"d8802708-e588-476f-b55b-bce1a3efa0cd","Type":"ContainerDied","Data":"788283d500347b6558b8e91370440c725f2ca322453e7f3ec4ed13bc3c56f175"}
Oct 30 05:00:19.653175 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.652617    5072 scope.go:122] "RemoveContainer" containerID="5cefd7387af4cc2d173da30cf9a5698eaba7de6ae0255561232cc8a2e50c53b3"
Oct 30 05:00:19.655327 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.655303    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk" event={"ID":"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61","Type":"ContainerStarted","Data":"559823f0cb04a5be759131db2d91c3feb1c91f50a6d7d56299a920cd13a7dda7"}
Oct 30 05:00:19.657938 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.657912    5072 generic.go:356] "Generic (PLEG): container finished" podID="92e82ab9-06b8-4c71-b46a-76348dee31b5" containerID="e90064d5e85ca2d22580eb16e5a6e8312d31a3d176dc65995b49ebb1d3faea41" exitCode=60
Oct 30 05:00:19.658160 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.657989    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb" event={"ID":"92e82ab9-06b8-4c71-b46a-76348dee31b5","Type":"ContainerDied","Data":"e90064d5e85ca2d22580eb16e5a6e8312d31a3d176dc65995b49ebb1d3faea41"}
Oct 30 05:00:19.658614 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.658594    5072 scope.go:122] "RemoveContainer" containerID="e90064d5e85ca2d22580eb16e5a6e8312d31a3d176dc65995b49ebb1d3faea41"
Oct 30 05:00:19.658989 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:19.658919    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb_odh-ai-gateway-infra(92e82ab9-06b8-4c71-b46a-76348dee31b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5"
Oct 30 05:00:19.660280 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.660260    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"
Oct 30 05:00:19.660559 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.660261    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c" event={"ID":"5d6f868c-5bea-49e5-a136-ab5862d1841d","Type":"ContainerDied","Data":"faa2a3a9ad07d050e6ae9b57e4e2f1fc973207a37d117b36a52d919131b20b1b"}
Oct 30 05:00:19.665427 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.665382    5072 generic.go:356] "Generic (PLEG): container finished" podID="f638f03d-08bb-4937-aa89-2198ac4a4c02" containerID="eca3494836a00696569f03a166adc2052d62c6fb21f856b5342c0b14fd9cd3fa" exitCode=60
Oct 30 05:00:19.665519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.665470    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g" event={"ID":"f638f03d-08bb-4937-aa89-2198ac4a4c02","Type":"ContainerDied","Data":"eca3494836a00696569f03a166adc2052d62c6fb21f856b5342c0b14fd9cd3fa"}
Oct 30 05:00:19.665875 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.665853    5072 scope.go:122] "RemoveContainer" containerID="eca3494836a00696569f03a166adc2052d62c6fb21f856b5342c0b14fd9cd3fa"
Oct 30 05:00:19.666305 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:19.666280    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g_odh-ai-gateway-infra(f638f03d-08bb-4937-aa89-2198ac4a4c02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02"
Oct 30 05:00:19.669971 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.669940    5072 generic.go:356] "Generic (PLEG): container finished" podID="804abcc6-7570-4f94-be22-d2c8b12f2108" containerID="48a156daf2fd4b9964073d5e870a1170237ef4322aa18a5514a6adb1f1894641" exitCode=60
Oct 30 05:00:19.670156 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.670030    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h" event={"ID":"804abcc6-7570-4f94-be22-d2c8b12f2108","Type":"ContainerDied","Data":"48a156daf2fd4b9964073d5e870a1170237ef4322aa18a5514a6adb1f1894641"}
Oct 30 05:00:19.670573 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.670556    5072 scope.go:122] "RemoveContainer" containerID="48a156daf2fd4b9964073d5e870a1170237ef4322aa18a5514a6adb1f1894641"
Oct 30 05:00:19.670833 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:19.670814    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h_odh-ai-gateway-infra(804abcc6-7570-4f94-be22-d2c8b12f2108)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108"
Oct 30 05:00:19.672605 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.672579    5072 generic.go:356] "Generic (PLEG): container finished" podID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" containerID="7c283dcebd94d37319c044a0a4258c2b7611ce3af474c3f3a9c60db60f64daa4" exitCode=60
Oct 30 05:00:19.672680 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.672640    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh" event={"ID":"7fdb04a5-7116-41a3-80cd-f1169b75cc2b","Type":"ContainerDied","Data":"7c283dcebd94d37319c044a0a4258c2b7611ce3af474c3f3a9c60db60f64daa4"}
Oct 30 05:00:19.672968 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.672952    5072 scope.go:122] "RemoveContainer" containerID="7c283dcebd94d37319c044a0a4258c2b7611ce3af474c3f3a9c60db60f64daa4"
Oct 30 05:00:19.673252 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:19.673230    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888940-bbnrh_odh-ai-gateway-infra(7fdb04a5-7116-41a3-80cd-f1169b75cc2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b"
Oct 30 05:00:19.674876 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.674852    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" event={"ID":"0e846f7c-186a-4e49-ba15-6ff6c564bc13","Type":"ContainerStarted","Data":"a31270a57f9cf4ed9e048ea0f8be3634adbc4a9027e6610071b1348e585f70ae"}
Oct 30 05:00:19.675216 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.675196    5072 scope.go:122] "RemoveContainer" containerID="a31270a57f9cf4ed9e048ea0f8be3634adbc4a9027e6610071b1348e585f70ae"
Oct 30 05:00:19.675438 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:19.675419    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6_odh-ai-gateway-infra(0e846f7c-186a-4e49-ba15-6ff6c564bc13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13"
Oct 30 05:00:19.677123 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.677064    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h" event={"ID":"06e8c6d6-1834-4ac0-a21d-998da69a4379","Type":"ContainerStarted","Data":"6d1fcc8d43b7c62e5bde0646e7c5702906799065d769f2bf7b88ce73f0743530"}
Oct 30 05:00:19.679050 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.679028    5072 generic.go:356] "Generic (PLEG): container finished" podID="7a540e59-8239-423a-ac80-aa84d7b4821d" containerID="15b7d378087542d122c41de65b56ca140b383b0d4475373db8c61be19f4f1f15" exitCode=60
Oct 30 05:00:19.679169 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.679126    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw" event={"ID":"7a540e59-8239-423a-ac80-aa84d7b4821d","Type":"ContainerDied","Data":"15b7d378087542d122c41de65b56ca140b383b0d4475373db8c61be19f4f1f15"}
Oct 30 05:00:19.679389 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.679376    5072 scope.go:122] "RemoveContainer" containerID="15b7d378087542d122c41de65b56ca140b383b0d4475373db8c61be19f4f1f15"
Oct 30 05:00:19.679573 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:19.679557    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw_odh-ai-gateway-infra(7a540e59-8239-423a-ac80-aa84d7b4821d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d"
Oct 30 05:00:19.737182 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.737015    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"]
Oct 30 05:00:19.747223 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.747190    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"]
Oct 30 05:00:19.773401 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.773343    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"]
Oct 30 05:00:19.860849 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.860820    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"]
Oct 30 05:00:19.935452 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.935431    5072 scope.go:122] "RemoveContainer" containerID="2138d1a10a60a072b9380a25ca7eafc48ad0145135688f95fbec566c89b5a11e"
Oct 30 05:00:19.967992 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.967968    5072 scope.go:122] "RemoveContainer" containerID="630388f646d3171193447aa2c61672d7ceed40606a008d6602a97a8d3f52864f"
Oct 30 05:00:19.984512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:19.984490    5072 scope.go:122] "RemoveContainer" containerID="51a84f959641959ea6068db972539d24fc2db2075adc2abda73be2f1ce19dcac"
Oct 30 05:00:20.000650 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.000627    5072 scope.go:122] "RemoveContainer" containerID="108e134f7eaf299304768554dcc88d15750c43815a1b9ea1780917ae70eefd03"
Oct 30 05:00:20.015877 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.015850    5072 scope.go:122] "RemoveContainer" containerID="edc316ab392a72d45495dc6cd1e7f2646dc3bb0b55a290dfd82527c0a3e1069a"
Oct 30 05:00:20.030645 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.030627    5072 scope.go:122] "RemoveContainer" containerID="d365486c30671c65b714592a62e9ed7113fb64bddae0406abafdd0568eb4fbd5"
Oct 30 05:00:20.119337 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.119307    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"]
Oct 30 05:00:20.124514 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.124489    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-483bea-29888940-m97zw"]
Oct 30 05:00:20.156641 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.156617    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"]
Oct 30 05:00:20.164770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.164748    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888940-rhq5c"]
Oct 30 05:00:20.407301 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.407218    5072 scope.go:122] "RemoveContainer" containerID="8b48e76ffc3256ff64f90aaf9f650b41e6f353ebebc176b861874f746cb85f2b"
Oct 30 05:00:20.407466 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.407324    5072 scope.go:122] "RemoveContainer" containerID="20d2ed910c168a11ba67f791a63dd0983d7a684d40f27cc9774cf9f5670d7040"
Oct 30 05:00:20.407466 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.407451    5072 scope.go:122] "RemoveContainer" containerID="7730d03d447b4940009fcce0c0bf7d439787909417eed32da0ca3e5cc5e5aa1d"
Oct 30 05:00:20.407628 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.407609    5072 scope.go:122] "RemoveContainer" containerID="39f56b4f616cb5c4e35758f5f531097cf015b3c4ef80f72fd7c903439504ab5a"
Oct 30 05:00:20.408297 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.408272    5072 scope.go:122] "RemoveContainer" containerID="041f5cd1a538fc86d15850849ca97c3e0a9c9d29ef4f332d6c5c3c5052ac8b53"
Oct 30 05:00:20.408782 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.408380    5072 scope.go:122] "RemoveContainer" containerID="013efd90b4d50e87f984bdb52e6bd7a24878a7ed7282b6baf07dc53d30a54ad2"
Oct 30 05:00:20.408782 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.408568    5072 scope.go:122] "RemoveContainer" containerID="11c3e6636e37e94b9b34a11c8ba06c213ec7881f2f8a9966e7afa509f06bf722"
Oct 30 05:00:20.414881 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.414854    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d" path="/var/lib/kubelet/pods/5d6f868c-5bea-49e5-a136-ab5862d1841d/volumes"
Oct 30 05:00:20.415457 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.415439    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd" path="/var/lib/kubelet/pods/d8802708-e588-476f-b55b-bce1a3efa0cd/volumes"
Oct 30 05:00:20.691142 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.691069    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9" event={"ID":"212e51ce-0a69-4809-822f-6f6fe6bf8773","Type":"ContainerStarted","Data":"ed4db2842b3fdfb4f31dbfb838c4504eb5947dc23df07ebb60d2c8f00ca4c5d4"}
Oct 30 05:00:20.693316 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.693289    5072 generic.go:356] "Generic (PLEG): container finished" podID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" containerID="a31270a57f9cf4ed9e048ea0f8be3634adbc4a9027e6610071b1348e585f70ae" exitCode=60
Oct 30 05:00:20.693455 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.693361    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" event={"ID":"0e846f7c-186a-4e49-ba15-6ff6c564bc13","Type":"ContainerDied","Data":"a31270a57f9cf4ed9e048ea0f8be3634adbc4a9027e6610071b1348e585f70ae"}
Oct 30 05:00:20.693455 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.693418    5072 scope.go:122] "RemoveContainer" containerID="b158b31edd6d6339c1588538cf92618a47b63783c2011f73cd72054dbaf5a737"
Oct 30 05:00:20.693802 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.693785    5072 scope.go:122] "RemoveContainer" containerID="a31270a57f9cf4ed9e048ea0f8be3634adbc4a9027e6610071b1348e585f70ae"
Oct 30 05:00:20.702473 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:20.694836    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6_odh-ai-gateway-infra(0e846f7c-186a-4e49-ba15-6ff6c564bc13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13"
Oct 30 05:00:20.708948 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.708925    5072 generic.go:356] "Generic (PLEG): container finished" podID="06e8c6d6-1834-4ac0-a21d-998da69a4379" containerID="6d1fcc8d43b7c62e5bde0646e7c5702906799065d769f2bf7b88ce73f0743530" exitCode=60
Oct 30 05:00:20.709126 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.709000    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h" event={"ID":"06e8c6d6-1834-4ac0-a21d-998da69a4379","Type":"ContainerDied","Data":"6d1fcc8d43b7c62e5bde0646e7c5702906799065d769f2bf7b88ce73f0743530"}
Oct 30 05:00:20.709414 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.709391    5072 scope.go:122] "RemoveContainer" containerID="6d1fcc8d43b7c62e5bde0646e7c5702906799065d769f2bf7b88ce73f0743530"
Oct 30 05:00:20.709794 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:20.709773    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h_odh-ai-gateway-infra(06e8c6d6-1834-4ac0-a21d-998da69a4379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379"
Oct 30 05:00:20.711245 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.711226    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9" event={"ID":"6fa98861-f2c2-47aa-bb83-54ddf8c5c715","Type":"ContainerStarted","Data":"23a02d899ce45c225195c53eb2479f5c10c83c844a0dea884db92602b9f09894"}
Oct 30 05:00:20.715263 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.715243    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n" event={"ID":"3621a9d6-ea49-415f-a6a9-5ce99245d111","Type":"ContainerStarted","Data":"b17b33c41e840b88e72414cbd8f9addc8f653774814c1210d74b4c509a0b6fe6"}
Oct 30 05:00:20.717405 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.717386    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2" event={"ID":"fc366c3b-6526-4575-899f-df191a5971de","Type":"ContainerStarted","Data":"eb59b27c558ff05dcb8e0d5428ec00d325b83f0f65b6a7471ed0d907f3b264ae"}
Oct 30 05:00:20.720221 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.720203    5072 scope.go:122] "RemoveContainer" containerID="578a7498bedac1175fd73411eddb28b00095e97be0a2d1b39201bb8d258d1294"
Oct 30 05:00:20.720573 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.720555    5072 generic.go:356] "Generic (PLEG): container finished" podID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" containerID="559823f0cb04a5be759131db2d91c3feb1c91f50a6d7d56299a920cd13a7dda7" exitCode=60
Oct 30 05:00:20.720651 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.720616    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk" event={"ID":"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61","Type":"ContainerDied","Data":"559823f0cb04a5be759131db2d91c3feb1c91f50a6d7d56299a920cd13a7dda7"}
Oct 30 05:00:20.720918 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.720900    5072 scope.go:122] "RemoveContainer" containerID="559823f0cb04a5be759131db2d91c3feb1c91f50a6d7d56299a920cd13a7dda7"
Oct 30 05:00:20.721183 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:20.721163    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk_odh-ai-gateway-infra(8dff5e9e-5890-4180-8c30-ee2b8a1b4e61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61"
Oct 30 05:00:20.731650 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.731628    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt" event={"ID":"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3","Type":"ContainerStarted","Data":"3f083f53651eaeddcc573eac3550c79298c2eae1e97902fd09498c09da9fb972"}
Oct 30 05:00:20.741190 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.741168    5072 scope.go:122] "RemoveContainer" containerID="9e990a2dd0ebc27338d0b1fb4c8d60135554438ebc178741dba87e83fd8bdb39"
Oct 30 05:00:20.828479 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.828441    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"]
Oct 30 05:00:20.935359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:20.934599    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"]
Oct 30 05:00:21.011552 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.011284    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"]
Oct 30 05:00:21.071783 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.066100    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"]
Oct 30 05:00:21.494636 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.494390    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"
Oct 30 05:00:21.501200 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.501176    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"
Oct 30 05:00:21.513778 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.513753    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"
Oct 30 05:00:21.527238 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.527214    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a540e59-8239-423a-ac80-aa84d7b4821d-kube-api-access-qmn5c\" (UniqueName: \"kubernetes.io/projected/7a540e59-8239-423a-ac80-aa84d7b4821d-kube-api-access-qmn5c\") pod \"7a540e59-8239-423a-ac80-aa84d7b4821d\" (UID: \"7a540e59-8239-423a-ac80-aa84d7b4821d\") "
Oct 30 05:00:21.527394 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.527313    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e82ab9-06b8-4c71-b46a-76348dee31b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e82ab9-06b8-4c71-b46a-76348dee31b5-maas-api-service-ca\") pod \"92e82ab9-06b8-4c71-b46a-76348dee31b5\" (UID: \"92e82ab9-06b8-4c71-b46a-76348dee31b5\") "
Oct 30 05:00:21.527394 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.527361    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e82ab9-06b8-4c71-b46a-76348dee31b5-kube-api-access-rdgsv\" (UniqueName: \"kubernetes.io/projected/92e82ab9-06b8-4c71-b46a-76348dee31b5-kube-api-access-rdgsv\") pod \"92e82ab9-06b8-4c71-b46a-76348dee31b5\" (UID: \"92e82ab9-06b8-4c71-b46a-76348dee31b5\") "
Oct 30 05:00:21.527510 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.527402    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a540e59-8239-423a-ac80-aa84d7b4821d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a540e59-8239-423a-ac80-aa84d7b4821d-maas-api-service-ca\") pod \"7a540e59-8239-423a-ac80-aa84d7b4821d\" (UID: \"7a540e59-8239-423a-ac80-aa84d7b4821d\") "
Oct 30 05:00:21.528232 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.528206    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e82ab9-06b8-4c71-b46a-76348dee31b5-maas-api-service-ca" pod "92e82ab9-06b8-4c71-b46a-76348dee31b5" (UID: "92e82ab9-06b8-4c71-b46a-76348dee31b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:21.528628 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.528605    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a540e59-8239-423a-ac80-aa84d7b4821d-maas-api-service-ca" pod "7a540e59-8239-423a-ac80-aa84d7b4821d" (UID: "7a540e59-8239-423a-ac80-aa84d7b4821d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:21.530651 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.530563    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e82ab9-06b8-4c71-b46a-76348dee31b5-kube-api-access-rdgsv" pod "92e82ab9-06b8-4c71-b46a-76348dee31b5" (UID: "92e82ab9-06b8-4c71-b46a-76348dee31b5"). InnerVolumeSpecName "kube-api-access-rdgsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:21.530755 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.530730    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a540e59-8239-423a-ac80-aa84d7b4821d-kube-api-access-qmn5c" pod "7a540e59-8239-423a-ac80-aa84d7b4821d" (UID: "7a540e59-8239-423a-ac80-aa84d7b4821d"). InnerVolumeSpecName "kube-api-access-qmn5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:21.580001 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.579865    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"
Oct 30 05:00:21.628615 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.628582    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804abcc6-7570-4f94-be22-d2c8b12f2108-kube-api-access-xqn8x\" (UniqueName: \"kubernetes.io/projected/804abcc6-7570-4f94-be22-d2c8b12f2108-kube-api-access-xqn8x\") pod \"804abcc6-7570-4f94-be22-d2c8b12f2108\" (UID: \"804abcc6-7570-4f94-be22-d2c8b12f2108\") "
Oct 30 05:00:21.628936 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.628911    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804abcc6-7570-4f94-be22-d2c8b12f2108-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/804abcc6-7570-4f94-be22-d2c8b12f2108-maas-api-service-ca\") pod \"804abcc6-7570-4f94-be22-d2c8b12f2108\" (UID: \"804abcc6-7570-4f94-be22-d2c8b12f2108\") "
Oct 30 05:00:21.629063 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.628969    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f638f03d-08bb-4937-aa89-2198ac4a4c02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f638f03d-08bb-4937-aa89-2198ac4a4c02-maas-api-service-ca\") pod \"f638f03d-08bb-4937-aa89-2198ac4a4c02\" (UID: \"f638f03d-08bb-4937-aa89-2198ac4a4c02\") "
Oct 30 05:00:21.629063 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.629016    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f638f03d-08bb-4937-aa89-2198ac4a4c02-kube-api-access-t5b4f\" (UniqueName: \"kubernetes.io/projected/f638f03d-08bb-4937-aa89-2198ac4a4c02-kube-api-access-t5b4f\") pod \"f638f03d-08bb-4937-aa89-2198ac4a4c02\" (UID: \"f638f03d-08bb-4937-aa89-2198ac4a4c02\") "
Oct 30 05:00:21.629362 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.629315    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804abcc6-7570-4f94-be22-d2c8b12f2108-maas-api-service-ca" pod "804abcc6-7570-4f94-be22-d2c8b12f2108" (UID: "804abcc6-7570-4f94-be22-d2c8b12f2108"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:21.629467 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.629381    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f638f03d-08bb-4937-aa89-2198ac4a4c02-maas-api-service-ca" pod "f638f03d-08bb-4937-aa89-2198ac4a4c02" (UID: "f638f03d-08bb-4937-aa89-2198ac4a4c02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:21.629741 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.629718    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmn5c\" (UniqueName: \"kubernetes.io/projected/7a540e59-8239-423a-ac80-aa84d7b4821d-kube-api-access-qmn5c\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:21.630157 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.629747    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/804abcc6-7570-4f94-be22-d2c8b12f2108-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:21.630157 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.629762    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f638f03d-08bb-4937-aa89-2198ac4a4c02-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:21.630157 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.629776    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e82ab9-06b8-4c71-b46a-76348dee31b5-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:21.630157 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.629789    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdgsv\" (UniqueName: \"kubernetes.io/projected/92e82ab9-06b8-4c71-b46a-76348dee31b5-kube-api-access-rdgsv\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:21.630157 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.629803    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a540e59-8239-423a-ac80-aa84d7b4821d-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:21.658019 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.657948    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f638f03d-08bb-4937-aa89-2198ac4a4c02-kube-api-access-t5b4f" pod "f638f03d-08bb-4937-aa89-2198ac4a4c02" (UID: "f638f03d-08bb-4937-aa89-2198ac4a4c02"). InnerVolumeSpecName "kube-api-access-t5b4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:21.658925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.658876    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804abcc6-7570-4f94-be22-d2c8b12f2108-kube-api-access-xqn8x" pod "804abcc6-7570-4f94-be22-d2c8b12f2108" (UID: "804abcc6-7570-4f94-be22-d2c8b12f2108"). InnerVolumeSpecName "kube-api-access-xqn8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:21.743402 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.742248    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"
Oct 30 05:00:21.744854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.744824    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqn8x\" (UniqueName: \"kubernetes.io/projected/804abcc6-7570-4f94-be22-d2c8b12f2108-kube-api-access-xqn8x\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:21.745190 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.745118    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5b4f\" (UniqueName: \"kubernetes.io/projected/f638f03d-08bb-4937-aa89-2198ac4a4c02-kube-api-access-t5b4f\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:21.770890 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.770850    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"]
Oct 30 05:00:21.773465 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.773434    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb" event={"ID":"92e82ab9-06b8-4c71-b46a-76348dee31b5","Type":"ContainerDied","Data":"1f54b7c8c4033eae462f21c1acb4efb72aeab038f0effc70bb2e8b2c20b55dc9"}
Oct 30 05:00:21.773601 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.773500    5072 scope.go:122] "RemoveContainer" containerID="e90064d5e85ca2d22580eb16e5a6e8312d31a3d176dc65995b49ebb1d3faea41"
Oct 30 05:00:21.773660 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.773604    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"
Oct 30 05:00:21.776848 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.776811    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs" event={"ID":"c9acd468-3d02-4521-9db7-33e9dff0259f","Type":"ContainerStarted","Data":"1c263d516a23f58da2df6c566c703f2c6262970fd440850dc555ad7719d26e4c"}
Oct 30 05:00:21.779574 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.779544    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g" event={"ID":"f638f03d-08bb-4937-aa89-2198ac4a4c02","Type":"ContainerDied","Data":"60a0c86cfadb947410136dbfb944ae60a0a5f93cd3e22d87887bd334055314ff"}
Oct 30 05:00:21.779684 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.779653    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"
Oct 30 05:00:21.794685 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.794650    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7" event={"ID":"82ca5f62-df90-42b8-91cb-dadacd5b3668","Type":"ContainerStarted","Data":"76d06d6483ef4c9a04196977108dfac33a8d8339dd82a957ceac4c54a195e03c"}
Oct 30 05:00:21.798125 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.797399    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"
Oct 30 05:00:21.798125 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.797407    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h" event={"ID":"804abcc6-7570-4f94-be22-d2c8b12f2108","Type":"ContainerDied","Data":"79d07a4e826d98be25f445ac78462cf98efa878e24fd85be3b9c0dbffd835534"}
Oct 30 05:00:21.804546 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.804523    5072 scope.go:122] "RemoveContainer" containerID="eca3494836a00696569f03a166adc2052d62c6fb21f856b5342c0b14fd9cd3fa"
Oct 30 05:00:21.806299 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.806272    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d" event={"ID":"a5cb00e5-a10c-4e5a-846b-7e10e452611e","Type":"ContainerStarted","Data":"4bb0331ab460c124dba98b9149172138bd1e611d49563b9669d085b95662303f"}
Oct 30 05:00:21.814642 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.814618    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"
Oct 30 05:00:21.816775 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.816683    5072 generic.go:356] "Generic (PLEG): container finished" podID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" containerID="3f083f53651eaeddcc573eac3550c79298c2eae1e97902fd09498c09da9fb972" exitCode=60
Oct 30 05:00:21.820165 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.813157    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh" event={"ID":"7fdb04a5-7116-41a3-80cd-f1169b75cc2b","Type":"ContainerDied","Data":"f799aa5f529902b321693e149153ab2e44b5b164ad90890d30946cbf05261bf3"}
Oct 30 05:00:21.820277 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.820204    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt" event={"ID":"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3","Type":"ContainerDied","Data":"3f083f53651eaeddcc573eac3550c79298c2eae1e97902fd09498c09da9fb972"}
Oct 30 05:00:21.820277 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.820234    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv" event={"ID":"f52b50b1-dfe7-4316-9918-61d44366cd60","Type":"ContainerStarted","Data":"5b5ad3e911c8cd7800b690d3405bf8ade1bd4665026717ed81f72da8617e363a"}
Oct 30 05:00:21.822450 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.821154    5072 scope.go:122] "RemoveContainer" containerID="3f083f53651eaeddcc573eac3550c79298c2eae1e97902fd09498c09da9fb972"
Oct 30 05:00:21.822450 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:21.822107    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt_odh-ai-gateway-infra(ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3"
Oct 30 05:00:21.830334 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.830304    5072 generic.go:356] "Generic (PLEG): container finished" podID="212e51ce-0a69-4809-822f-6f6fe6bf8773" containerID="ed4db2842b3fdfb4f31dbfb838c4504eb5947dc23df07ebb60d2c8f00ca4c5d4" exitCode=60
Oct 30 05:00:21.830455 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.830416    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9" event={"ID":"212e51ce-0a69-4809-822f-6f6fe6bf8773","Type":"ContainerDied","Data":"ed4db2842b3fdfb4f31dbfb838c4504eb5947dc23df07ebb60d2c8f00ca4c5d4"}
Oct 30 05:00:21.834855 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.834830    5072 scope.go:122] "RemoveContainer" containerID="48a156daf2fd4b9964073d5e870a1170237ef4322aa18a5514a6adb1f1894641"
Oct 30 05:00:21.835407 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.835266    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"]
Oct 30 05:00:21.857221 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.845759    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf" event={"ID":"2a82ff41-ac42-48bc-81c4-24e539017282","Type":"ContainerStarted","Data":"30ca9bd881938f1a754802dc49a62fcd0bdda8fc1ab0f6f27ecfb665f57e9927"}
Oct 30 05:00:21.857221 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.848316    5072 generic.go:356] "Generic (PLEG): container finished" podID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" containerID="23a02d899ce45c225195c53eb2479f5c10c83c844a0dea884db92602b9f09894" exitCode=60
Oct 30 05:00:21.857221 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.848407    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9" event={"ID":"6fa98861-f2c2-47aa-bb83-54ddf8c5c715","Type":"ContainerDied","Data":"23a02d899ce45c225195c53eb2479f5c10c83c844a0dea884db92602b9f09894"}
Oct 30 05:00:21.888373 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.875359    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"
Oct 30 05:00:21.888373 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.872066    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5" event={"ID":"e53d6987-db5e-468a-9afe-2c0603cbf408","Type":"ContainerStarted","Data":"915c47259b17c9c3c2e88d429c38ab1878306612eee9475eeb617d784e291376"}
Oct 30 05:00:21.888373 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.877789    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw" event={"ID":"7a540e59-8239-423a-ac80-aa84d7b4821d","Type":"ContainerDied","Data":"5422d3478931f9532833882e65891fc2e3b4636600cac4206891a9855f019b3b"}
Oct 30 05:00:21.888373 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.881199    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m" event={"ID":"927eaf22-dd8b-4bef-a48c-fff361ead463","Type":"ContainerStarted","Data":"75003dc0e4ca30df3d3d792e62520746a3c18c8399a45154150d47a6536655f4"}
Oct 30 05:00:21.899345 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.891720    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-kube-api-access-68kcr\" (UniqueName: \"kubernetes.io/projected/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-kube-api-access-68kcr\") pod \"7fdb04a5-7116-41a3-80cd-f1169b75cc2b\" (UID: \"7fdb04a5-7116-41a3-80cd-f1169b75cc2b\") "
Oct 30 05:00:21.899345 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.892160    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-maas-api-service-ca\") pod \"7fdb04a5-7116-41a3-80cd-f1169b75cc2b\" (UID: \"7fdb04a5-7116-41a3-80cd-f1169b75cc2b\") "
Oct 30 05:00:21.899345 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.893589    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-maas-api-service-ca" pod "7fdb04a5-7116-41a3-80cd-f1169b75cc2b" (UID: "7fdb04a5-7116-41a3-80cd-f1169b75cc2b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:21.899345 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.898647    5072 scope.go:122] "RemoveContainer" containerID="7c283dcebd94d37319c044a0a4258c2b7611ce3af474c3f3a9c60db60f64daa4"
Oct 30 05:00:21.899767 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.899702    5072 generic.go:356] "Generic (PLEG): container finished" podID="3621a9d6-ea49-415f-a6a9-5ce99245d111" containerID="b17b33c41e840b88e72414cbd8f9addc8f653774814c1210d74b4c509a0b6fe6" exitCode=60
Oct 30 05:00:21.901918 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.899817    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n" event={"ID":"3621a9d6-ea49-415f-a6a9-5ce99245d111","Type":"ContainerDied","Data":"b17b33c41e840b88e72414cbd8f9addc8f653774814c1210d74b4c509a0b6fe6"}
Oct 30 05:00:21.901918 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.900216    5072 scope.go:122] "RemoveContainer" containerID="b17b33c41e840b88e72414cbd8f9addc8f653774814c1210d74b4c509a0b6fe6"
Oct 30 05:00:21.901918 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:21.900489    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n_odh-ai-gateway-infra(3621a9d6-ea49-415f-a6a9-5ce99245d111)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111"
Oct 30 05:00:21.917561 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.914041    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"]
Oct 30 05:00:21.925139 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.919278    5072 generic.go:356] "Generic (PLEG): container finished" podID="fc366c3b-6526-4575-899f-df191a5971de" containerID="eb59b27c558ff05dcb8e0d5428ec00d325b83f0f65b6a7471ed0d907f3b264ae" exitCode=60
Oct 30 05:00:21.925139 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.920636    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2" event={"ID":"fc366c3b-6526-4575-899f-df191a5971de","Type":"ContainerDied","Data":"eb59b27c558ff05dcb8e0d5428ec00d325b83f0f65b6a7471ed0d907f3b264ae"}
Oct 30 05:00:21.925139 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.920695    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"]
Oct 30 05:00:21.925139 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.921227    5072 scope.go:122] "RemoveContainer" containerID="eb59b27c558ff05dcb8e0d5428ec00d325b83f0f65b6a7471ed0d907f3b264ae"
Oct 30 05:00:21.925139 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:21.921552    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2_odh-ai-gateway-infra(fc366c3b-6526-4575-899f-df191a5971de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2" podUID="fc366c3b-6526-4575-899f-df191a5971de"
Oct 30 05:00:21.933494 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.933011    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449d83-29888940-pm6sb"]
Oct 30 05:00:21.969344 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.969290    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-kube-api-access-68kcr" pod "7fdb04a5-7116-41a3-80cd-f1169b75cc2b" (UID: "7fdb04a5-7116-41a3-80cd-f1169b75cc2b"). InnerVolumeSpecName "kube-api-access-68kcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:21.992718 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.992686    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"]
Oct 30 05:00:21.998055 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.997979    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68kcr\" (UniqueName: \"kubernetes.io/projected/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-kube-api-access-68kcr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:21.998055 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:21.998005    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdb04a5-7116-41a3-80cd-f1169b75cc2b-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:22.067238 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.067137    5072 scope.go:122] "RemoveContainer" containerID="3f28bad762c18973802882cd6d7ddc0cc873b5d8424ff23e799ca911093e30c0"
Oct 30 05:00:22.163406 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.163373    5072 scope.go:122] "RemoveContainer" containerID="27475a32b37d513229f67e0d473cf5a4f1d45659579df7ace12a3212c32a8d56"
Oct 30 05:00:22.172636 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.172611    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"
Oct 30 05:00:22.190434 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.190403    5072 scope.go:122] "RemoveContainer" containerID="16e10d0b9c9bd56da65bf95af053716430b0dbf497d5bc33241e767f8b09bf49"
Oct 30 05:00:22.211194 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.211150    5072 scope.go:122] "RemoveContainer" containerID="15b7d378087542d122c41de65b56ca140b383b0d4475373db8c61be19f4f1f15"
Oct 30 05:00:22.219965 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.219944    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"]
Oct 30 05:00:22.249253 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.248011    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"]
Oct 30 05:00:22.249253 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.248063    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5d74b9-29888940-9ls8g"]
Oct 30 05:00:22.253375 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.252293    5072 scope.go:122] "RemoveContainer" containerID="591a426acee69061f04dd12a71dab7b97f9e69b1b4128d3de05b4312c743be24"
Oct 30 05:00:22.299372 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.299344    5072 scope.go:122] "RemoveContainer" containerID="da2e437e983141ec02fa7b21035f90ccec5f0efd176f4dd790833fabb8902c5e"
Oct 30 05:00:22.307744 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.307718    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-kube-api-access-zb9vw\" (UniqueName: \"kubernetes.io/projected/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-kube-api-access-zb9vw\") pod \"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61\" (UID: \"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61\") "
Oct 30 05:00:22.308049 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.308024    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-maas-api-service-ca\") pod \"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61\" (UID: \"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61\") "
Oct 30 05:00:22.308710 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.308415    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-maas-api-service-ca" pod "8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" (UID: "8dff5e9e-5890-4180-8c30-ee2b8a1b4e61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:22.309247 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.309210    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:22.310736 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.310642    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-kube-api-access-zb9vw" pod "8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" (UID: "8dff5e9e-5890-4180-8c30-ee2b8a1b4e61"). InnerVolumeSpecName "kube-api-access-zb9vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:22.316618 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.316597    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"
Oct 30 05:00:22.322345 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.322321    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"
Oct 30 05:00:22.326728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.326710    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"
Oct 30 05:00:22.331174 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.331156    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"
Oct 30 05:00:22.377228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.377199    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"]
Oct 30 05:00:22.382598 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.382573    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f9cd17-29888940-vx47h"]
Oct 30 05:00:22.410270 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.410241    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212e51ce-0a69-4809-822f-6f6fe6bf8773-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212e51ce-0a69-4809-822f-6f6fe6bf8773-maas-api-service-ca\") pod \"212e51ce-0a69-4809-822f-6f6fe6bf8773\" (UID: \"212e51ce-0a69-4809-822f-6f6fe6bf8773\") "
Oct 30 05:00:22.410396 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.410364    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e846f7c-186a-4e49-ba15-6ff6c564bc13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e846f7c-186a-4e49-ba15-6ff6c564bc13-maas-api-service-ca\") pod \"0e846f7c-186a-4e49-ba15-6ff6c564bc13\" (UID: \"0e846f7c-186a-4e49-ba15-6ff6c564bc13\") "
Oct 30 05:00:22.410437 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.410419    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-kube-api-access-48d64\" (UniqueName: \"kubernetes.io/projected/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-kube-api-access-48d64\") pod \"6fa98861-f2c2-47aa-bb83-54ddf8c5c715\" (UID: \"6fa98861-f2c2-47aa-bb83-54ddf8c5c715\") "
Oct 30 05:00:22.410503 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.410482    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-maas-api-service-ca\") pod \"6fa98861-f2c2-47aa-bb83-54ddf8c5c715\" (UID: \"6fa98861-f2c2-47aa-bb83-54ddf8c5c715\") "
Oct 30 05:00:22.410571 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.410554    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212e51ce-0a69-4809-822f-6f6fe6bf8773-kube-api-access-s4pqf\" (UniqueName: \"kubernetes.io/projected/212e51ce-0a69-4809-822f-6f6fe6bf8773-kube-api-access-s4pqf\") pod \"212e51ce-0a69-4809-822f-6f6fe6bf8773\" (UID: \"212e51ce-0a69-4809-822f-6f6fe6bf8773\") "
Oct 30 05:00:22.410622 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.410603    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e846f7c-186a-4e49-ba15-6ff6c564bc13-kube-api-access-4kd87\" (UniqueName: \"kubernetes.io/projected/0e846f7c-186a-4e49-ba15-6ff6c564bc13-kube-api-access-4kd87\") pod \"0e846f7c-186a-4e49-ba15-6ff6c564bc13\" (UID: \"0e846f7c-186a-4e49-ba15-6ff6c564bc13\") "
Oct 30 05:00:22.410695 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.410678    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e8c6d6-1834-4ac0-a21d-998da69a4379-kube-api-access-dtnw6\" (UniqueName: \"kubernetes.io/projected/06e8c6d6-1834-4ac0-a21d-998da69a4379-kube-api-access-dtnw6\") pod \"06e8c6d6-1834-4ac0-a21d-998da69a4379\" (UID: \"06e8c6d6-1834-4ac0-a21d-998da69a4379\") "
Oct 30 05:00:22.410762 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.410738    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212e51ce-0a69-4809-822f-6f6fe6bf8773-maas-api-service-ca" pod "212e51ce-0a69-4809-822f-6f6fe6bf8773" (UID: "212e51ce-0a69-4809-822f-6f6fe6bf8773"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:22.410762 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.410754    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e8c6d6-1834-4ac0-a21d-998da69a4379-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e8c6d6-1834-4ac0-a21d-998da69a4379-maas-api-service-ca\") pod \"06e8c6d6-1834-4ac0-a21d-998da69a4379\" (UID: \"06e8c6d6-1834-4ac0-a21d-998da69a4379\") "
Oct 30 05:00:22.410879 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.410810    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e846f7c-186a-4e49-ba15-6ff6c564bc13-maas-api-service-ca" pod "0e846f7c-186a-4e49-ba15-6ff6c564bc13" (UID: "0e846f7c-186a-4e49-ba15-6ff6c564bc13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:22.411231 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.411208    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-maas-api-service-ca" pod "6fa98861-f2c2-47aa-bb83-54ddf8c5c715" (UID: "6fa98861-f2c2-47aa-bb83-54ddf8c5c715"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:22.411507 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.411215    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e8c6d6-1834-4ac0-a21d-998da69a4379-maas-api-service-ca" pod "06e8c6d6-1834-4ac0-a21d-998da69a4379" (UID: "06e8c6d6-1834-4ac0-a21d-998da69a4379"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:22.411760 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.411739    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212e51ce-0a69-4809-822f-6f6fe6bf8773-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:22.411844 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.411770    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e846f7c-186a-4e49-ba15-6ff6c564bc13-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:22.411844 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.411787    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:22.411844 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.411803    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb9vw\" (UniqueName: \"kubernetes.io/projected/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61-kube-api-access-zb9vw\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:22.411844 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.411817    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e8c6d6-1834-4ac0-a21d-998da69a4379-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:22.413210 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.413188    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108" path="/var/lib/kubelet/pods/804abcc6-7570-4f94-be22-d2c8b12f2108/volumes"
Oct 30 05:00:22.413657 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.413636    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5" path="/var/lib/kubelet/pods/92e82ab9-06b8-4c71-b46a-76348dee31b5/volumes"
Oct 30 05:00:22.414065 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.414050    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02" path="/var/lib/kubelet/pods/f638f03d-08bb-4937-aa89-2198ac4a4c02/volumes"
Oct 30 05:00:22.458326 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.457023    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e846f7c-186a-4e49-ba15-6ff6c564bc13-kube-api-access-4kd87" pod "0e846f7c-186a-4e49-ba15-6ff6c564bc13" (UID: "0e846f7c-186a-4e49-ba15-6ff6c564bc13"). InnerVolumeSpecName "kube-api-access-4kd87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:22.458326 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.457704    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212e51ce-0a69-4809-822f-6f6fe6bf8773-kube-api-access-s4pqf" pod "212e51ce-0a69-4809-822f-6f6fe6bf8773" (UID: "212e51ce-0a69-4809-822f-6f6fe6bf8773"). InnerVolumeSpecName "kube-api-access-s4pqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:22.458326 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.457808    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-kube-api-access-48d64" pod "6fa98861-f2c2-47aa-bb83-54ddf8c5c715" (UID: "6fa98861-f2c2-47aa-bb83-54ddf8c5c715"). InnerVolumeSpecName "kube-api-access-48d64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:22.458604 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.458516    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e8c6d6-1834-4ac0-a21d-998da69a4379-kube-api-access-dtnw6" pod "06e8c6d6-1834-4ac0-a21d-998da69a4379" (UID: "06e8c6d6-1834-4ac0-a21d-998da69a4379"). InnerVolumeSpecName "kube-api-access-dtnw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:22.513998 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.513931    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48d64\" (UniqueName: \"kubernetes.io/projected/6fa98861-f2c2-47aa-bb83-54ddf8c5c715-kube-api-access-48d64\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:22.513998 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.513962    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4pqf\" (UniqueName: \"kubernetes.io/projected/212e51ce-0a69-4809-822f-6f6fe6bf8773-kube-api-access-s4pqf\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:22.513998 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.513978    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kd87\" (UniqueName: \"kubernetes.io/projected/0e846f7c-186a-4e49-ba15-6ff6c564bc13-kube-api-access-4kd87\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:22.513998 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.513993    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtnw6\" (UniqueName: \"kubernetes.io/projected/06e8c6d6-1834-4ac0-a21d-998da69a4379-kube-api-access-dtnw6\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:22.629727 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.629670    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"]
Oct 30 05:00:22.636209 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.636179    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e6b837-29888940-ffzfw"]
Oct 30 05:00:22.811119 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.811030    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"]
Oct 30 05:00:22.818309 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.818276    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888940-bbnrh"]
Oct 30 05:00:22.875219 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.875188    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"]
Oct 30 05:00:22.942803 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.942768    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"
Oct 30 05:00:22.942981 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.942801    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9" event={"ID":"212e51ce-0a69-4809-822f-6f6fe6bf8773","Type":"ContainerDied","Data":"c5ff0fcb8f36279fe3dad03a77915c7ad8c5ae4f7704c7f26f1169a916fd2122"}
Oct 30 05:00:22.942981 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.942864    5072 scope.go:122] "RemoveContainer" containerID="ed4db2842b3fdfb4f31dbfb838c4504eb5947dc23df07ebb60d2c8f00ca4c5d4"
Oct 30 05:00:22.944773 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.944700    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"
Oct 30 05:00:22.944773 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.944711    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6" event={"ID":"0e846f7c-186a-4e49-ba15-6ff6c564bc13","Type":"ContainerDied","Data":"40006da4867c7690362e750a151751eaa6cc65e45baf581a0a9ffe5b6bd9311e"}
Oct 30 05:00:22.946331 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.946300    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"
Oct 30 05:00:22.946331 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.946324    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h" event={"ID":"06e8c6d6-1834-4ac0-a21d-998da69a4379","Type":"ContainerDied","Data":"f131b2276fbfa62ce6f1a70d029fecdc57e56d4a0b8c2ab160cc3be03fceb437"}
Oct 30 05:00:22.949179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.949157    5072 generic.go:356] "Generic (PLEG): container finished" podID="2a82ff41-ac42-48bc-81c4-24e539017282" containerID="30ca9bd881938f1a754802dc49a62fcd0bdda8fc1ab0f6f27ecfb665f57e9927" exitCode=60
Oct 30 05:00:22.949530 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.949316    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf" event={"ID":"2a82ff41-ac42-48bc-81c4-24e539017282","Type":"ContainerDied","Data":"30ca9bd881938f1a754802dc49a62fcd0bdda8fc1ab0f6f27ecfb665f57e9927"}
Oct 30 05:00:22.949896 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.949880    5072 scope.go:122] "RemoveContainer" containerID="30ca9bd881938f1a754802dc49a62fcd0bdda8fc1ab0f6f27ecfb665f57e9927"
Oct 30 05:00:22.950368 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:22.950158    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf_odh-ai-gateway-infra(2a82ff41-ac42-48bc-81c4-24e539017282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf" podUID="2a82ff41-ac42-48bc-81c4-24e539017282"
Oct 30 05:00:22.951401 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.951378    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9" event={"ID":"6fa98861-f2c2-47aa-bb83-54ddf8c5c715","Type":"ContainerDied","Data":"0a746d3413376925fca781dc2f81e969d66a33faf0f37f72de7336ceb5544004"}
Oct 30 05:00:22.951474 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.951463    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"
Oct 30 05:00:22.966722 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.966701    5072 scope.go:122] "RemoveContainer" containerID="a31270a57f9cf4ed9e048ea0f8be3634adbc4a9027e6610071b1348e585f70ae"
Oct 30 05:00:22.971053 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.971019    5072 generic.go:356] "Generic (PLEG): container finished" podID="e53d6987-db5e-468a-9afe-2c0603cbf408" containerID="915c47259b17c9c3c2e88d429c38ab1878306612eee9475eeb617d784e291376" exitCode=60
Oct 30 05:00:22.971177 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.971067    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5" event={"ID":"e53d6987-db5e-468a-9afe-2c0603cbf408","Type":"ContainerDied","Data":"915c47259b17c9c3c2e88d429c38ab1878306612eee9475eeb617d784e291376"}
Oct 30 05:00:22.971999 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.971980    5072 scope.go:122] "RemoveContainer" containerID="915c47259b17c9c3c2e88d429c38ab1878306612eee9475eeb617d784e291376"
Oct 30 05:00:22.972411 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:22.972386    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5_odh-ai-gateway-infra(e53d6987-db5e-468a-9afe-2c0603cbf408)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408"
Oct 30 05:00:22.976420 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.976281    5072 generic.go:356] "Generic (PLEG): container finished" podID="927eaf22-dd8b-4bef-a48c-fff361ead463" containerID="75003dc0e4ca30df3d3d792e62520746a3c18c8399a45154150d47a6536655f4" exitCode=60
Oct 30 05:00:22.977437 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.976295    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m" event={"ID":"927eaf22-dd8b-4bef-a48c-fff361ead463","Type":"ContainerDied","Data":"75003dc0e4ca30df3d3d792e62520746a3c18c8399a45154150d47a6536655f4"}
Oct 30 05:00:22.977437 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.976962    5072 scope.go:122] "RemoveContainer" containerID="75003dc0e4ca30df3d3d792e62520746a3c18c8399a45154150d47a6536655f4"
Oct 30 05:00:22.977437 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:22.977388    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m_odh-ai-gateway-infra(927eaf22-dd8b-4bef-a48c-fff361ead463)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463"
Oct 30 05:00:22.984912 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.984889    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk" event={"ID":"8dff5e9e-5890-4180-8c30-ee2b8a1b4e61","Type":"ContainerDied","Data":"6cd971112f1392c75fdf8cee40058edadc236f1c1e76b8e66579d0bea28144e8"}
Oct 30 05:00:22.985433 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.985414    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"
Oct 30 05:00:22.989670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.989650    5072 generic.go:356] "Generic (PLEG): container finished" podID="c9acd468-3d02-4521-9db7-33e9dff0259f" containerID="1c263d516a23f58da2df6c566c703f2c6262970fd440850dc555ad7719d26e4c" exitCode=60
Oct 30 05:00:22.989814 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.989796    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs" event={"ID":"c9acd468-3d02-4521-9db7-33e9dff0259f","Type":"ContainerDied","Data":"1c263d516a23f58da2df6c566c703f2c6262970fd440850dc555ad7719d26e4c"}
Oct 30 05:00:22.995537 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.994637    5072 generic.go:356] "Generic (PLEG): container finished" podID="82ca5f62-df90-42b8-91cb-dadacd5b3668" containerID="76d06d6483ef4c9a04196977108dfac33a8d8339dd82a957ceac4c54a195e03c" exitCode=60
Oct 30 05:00:22.995537 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.994703    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7" event={"ID":"82ca5f62-df90-42b8-91cb-dadacd5b3668","Type":"ContainerDied","Data":"76d06d6483ef4c9a04196977108dfac33a8d8339dd82a957ceac4c54a195e03c"}
Oct 30 05:00:22.995537 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:22.995110    5072 scope.go:122] "RemoveContainer" containerID="76d06d6483ef4c9a04196977108dfac33a8d8339dd82a957ceac4c54a195e03c"
Oct 30 05:00:22.995537 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:22.995478    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7_odh-ai-gateway-infra(82ca5f62-df90-42b8-91cb-dadacd5b3668)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668"
Oct 30 05:00:23.036254 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.036218    5072 scope.go:122] "RemoveContainer" containerID="6d1fcc8d43b7c62e5bde0646e7c5702906799065d769f2bf7b88ce73f0743530"
Oct 30 05:00:23.050617 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.050529    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"]
Oct 30 05:00:23.052671 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.051683    5072 generic.go:356] "Generic (PLEG): container finished" podID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" containerID="4bb0331ab460c124dba98b9149172138bd1e611d49563b9669d085b95662303f" exitCode=60
Oct 30 05:00:23.052671 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.051791    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d" event={"ID":"a5cb00e5-a10c-4e5a-846b-7e10e452611e","Type":"ContainerDied","Data":"4bb0331ab460c124dba98b9149172138bd1e611d49563b9669d085b95662303f"}
Oct 30 05:00:23.052671 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.052229    5072 scope.go:122] "RemoveContainer" containerID="4bb0331ab460c124dba98b9149172138bd1e611d49563b9669d085b95662303f"
Oct 30 05:00:23.052671 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:23.052603    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d_odh-ai-gateway-infra(a5cb00e5-a10c-4e5a-846b-7e10e452611e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e"
Oct 30 05:00:23.113107 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.111949    5072 generic.go:356] "Generic (PLEG): container finished" podID="f52b50b1-dfe7-4316-9918-61d44366cd60" containerID="5b5ad3e911c8cd7800b690d3405bf8ade1bd4665026717ed81f72da8617e363a" exitCode=60
Oct 30 05:00:23.113107 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.111996    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv" event={"ID":"f52b50b1-dfe7-4316-9918-61d44366cd60","Type":"ContainerDied","Data":"5b5ad3e911c8cd7800b690d3405bf8ade1bd4665026717ed81f72da8617e363a"}
Oct 30 05:00:23.113107 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.112501    5072 scope.go:122] "RemoveContainer" containerID="5b5ad3e911c8cd7800b690d3405bf8ade1bd4665026717ed81f72da8617e363a"
Oct 30 05:00:23.113107 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:23.112927    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv_odh-ai-gateway-infra(f52b50b1-dfe7-4316-9918-61d44366cd60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60"
Oct 30 05:00:23.192303 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.189996    5072 scope.go:122] "RemoveContainer" containerID="20d2ed910c168a11ba67f791a63dd0983d7a684d40f27cc9774cf9f5670d7040"
Oct 30 05:00:23.320589 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.320527    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"
Oct 30 05:00:23.346635 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.345609    5072 scope.go:122] "RemoveContainer" containerID="23a02d899ce45c225195c53eb2479f5c10c83c844a0dea884db92602b9f09894"
Oct 30 05:00:23.368307 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.368062    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"]
Oct 30 05:00:23.378161 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.378061    5072 scope.go:122] "RemoveContainer" containerID="8b48e76ffc3256ff64f90aaf9f650b41e6f353ebebc176b861874f746cb85f2b"
Oct 30 05:00:23.402800 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.402748    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"]
Oct 30 05:00:23.410665 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.410617    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-05bee8-29888940-gclqk"]
Oct 30 05:00:23.425421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.425399    5072 scope.go:122] "RemoveContainer" containerID="11c3e6636e37e94b9b34a11c8ba06c213ec7881f2f8a9966e7afa509f06bf722"
Oct 30 05:00:23.443335 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.443285    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc366c3b-6526-4575-899f-df191a5971de-kube-api-access-csgjj\" (UniqueName: \"kubernetes.io/projected/fc366c3b-6526-4575-899f-df191a5971de-kube-api-access-csgjj\") pod \"fc366c3b-6526-4575-899f-df191a5971de\" (UID: \"fc366c3b-6526-4575-899f-df191a5971de\") "
Oct 30 05:00:23.443568 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.443552    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc366c3b-6526-4575-899f-df191a5971de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc366c3b-6526-4575-899f-df191a5971de-maas-api-service-ca\") pod \"fc366c3b-6526-4575-899f-df191a5971de\" (UID: \"fc366c3b-6526-4575-899f-df191a5971de\") "
Oct 30 05:00:23.444169 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.444143    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc366c3b-6526-4575-899f-df191a5971de-maas-api-service-ca" pod "fc366c3b-6526-4575-899f-df191a5971de" (UID: "fc366c3b-6526-4575-899f-df191a5971de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:23.446137 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.446116    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc366c3b-6526-4575-899f-df191a5971de-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:23.484463 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.484420    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc366c3b-6526-4575-899f-df191a5971de-kube-api-access-csgjj" pod "fc366c3b-6526-4575-899f-df191a5971de" (UID: "fc366c3b-6526-4575-899f-df191a5971de"). InnerVolumeSpecName "kube-api-access-csgjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:23.505732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.505198    5072 scope.go:122] "RemoveContainer" containerID="559823f0cb04a5be759131db2d91c3feb1c91f50a6d7d56299a920cd13a7dda7"
Oct 30 05:00:23.548272 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.548242    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csgjj\" (UniqueName: \"kubernetes.io/projected/fc366c3b-6526-4575-899f-df191a5971de-kube-api-access-csgjj\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:23.556809 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.556772    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"]
Oct 30 05:00:23.580798 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.580773    5072 scope.go:122] "RemoveContainer" containerID="013efd90b4d50e87f984bdb52e6bd7a24878a7ed7282b6baf07dc53d30a54ad2"
Oct 30 05:00:23.595423 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.595391    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"]
Oct 30 05:00:23.616746 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.616720    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"]
Oct 30 05:00:23.758918 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.758872    5072 scope.go:122] "RemoveContainer" containerID="7730d03d447b4940009fcce0c0bf7d439787909417eed32da0ca3e5cc5e5aa1d"
Oct 30 05:00:23.762626 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.762405    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9f158a-29888940-t664h"]
Oct 30 05:00:23.779104 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.777330    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"
Oct 30 05:00:23.820371 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.820341    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"
Oct 30 05:00:23.886888 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.886679    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"]
Oct 30 05:00:23.909880 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.906582    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fcf42a-29888940-d7tp9"]
Oct 30 05:00:23.914019 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.913989    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"]
Oct 30 05:00:23.922143 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.922121    5072 scope.go:122] "RemoveContainer" containerID="041f5cd1a538fc86d15850849ca97c3e0a9c9d29ef4f332d6c5c3c5052ac8b53"
Oct 30 05:00:23.931262 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.930303    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"
Oct 30 05:00:23.936404 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.936328    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"]
Oct 30 05:00:23.940406 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.940384    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9acd468-3d02-4521-9db7-33e9dff0259f-kube-api-access-vcvhb\" (UniqueName: \"kubernetes.io/projected/c9acd468-3d02-4521-9db7-33e9dff0259f-kube-api-access-vcvhb\") pod \"c9acd468-3d02-4521-9db7-33e9dff0259f\" (UID: \"c9acd468-3d02-4521-9db7-33e9dff0259f\") "
Oct 30 05:00:23.941138 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.941118    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3621a9d6-ea49-415f-a6a9-5ce99245d111-kube-api-access-4vbgb\" (UniqueName: \"kubernetes.io/projected/3621a9d6-ea49-415f-a6a9-5ce99245d111-kube-api-access-4vbgb\") pod \"3621a9d6-ea49-415f-a6a9-5ce99245d111\" (UID: \"3621a9d6-ea49-415f-a6a9-5ce99245d111\") "
Oct 30 05:00:23.941300 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.941289    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3621a9d6-ea49-415f-a6a9-5ce99245d111-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3621a9d6-ea49-415f-a6a9-5ce99245d111-maas-api-service-ca\") pod \"3621a9d6-ea49-415f-a6a9-5ce99245d111\" (UID: \"3621a9d6-ea49-415f-a6a9-5ce99245d111\") "
Oct 30 05:00:23.941396 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.941386    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9acd468-3d02-4521-9db7-33e9dff0259f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9acd468-3d02-4521-9db7-33e9dff0259f-maas-api-service-ca\") pod \"c9acd468-3d02-4521-9db7-33e9dff0259f\" (UID: \"c9acd468-3d02-4521-9db7-33e9dff0259f\") "
Oct 30 05:00:23.942662 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.942618    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3621a9d6-ea49-415f-a6a9-5ce99245d111-maas-api-service-ca" pod "3621a9d6-ea49-415f-a6a9-5ce99245d111" (UID: "3621a9d6-ea49-415f-a6a9-5ce99245d111"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:23.943290 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.943268    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9acd468-3d02-4521-9db7-33e9dff0259f-maas-api-service-ca" pod "c9acd468-3d02-4521-9db7-33e9dff0259f" (UID: "c9acd468-3d02-4521-9db7-33e9dff0259f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:23.965157 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.961209    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9acd468-3d02-4521-9db7-33e9dff0259f-kube-api-access-vcvhb" pod "c9acd468-3d02-4521-9db7-33e9dff0259f" (UID: "c9acd468-3d02-4521-9db7-33e9dff0259f"). InnerVolumeSpecName "kube-api-access-vcvhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:23.988045 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.988012    5072 scope.go:122] "RemoveContainer" containerID="39f56b4f616cb5c4e35758f5f531097cf015b3c4ef80f72fd7c903439504ab5a"
Oct 30 05:00:24.000507 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:23.997785    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3621a9d6-ea49-415f-a6a9-5ce99245d111-kube-api-access-4vbgb" pod "3621a9d6-ea49-415f-a6a9-5ce99245d111" (UID: "3621a9d6-ea49-415f-a6a9-5ce99245d111"). InnerVolumeSpecName "kube-api-access-4vbgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:24.019200 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.019158    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"]
Oct 30 05:00:24.024013 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.023948    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134dad-29888940-hhpl9"]
Oct 30 05:00:24.027766 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.027741    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"]
Oct 30 05:00:24.034760 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.034730    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888940-csrg6"]
Oct 30 05:00:24.043565 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.043524    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-maas-api-service-ca\") pod \"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3\" (UID: \"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3\") "
Oct 30 05:00:24.043824 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.043801    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-kube-api-access-pnvh6\" (UniqueName: \"kubernetes.io/projected/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-kube-api-access-pnvh6\") pod \"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3\" (UID: \"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3\") "
Oct 30 05:00:24.044291 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.044136    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-maas-api-service-ca" pod "ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" (UID: "ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:24.044612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.044473    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcvhb\" (UniqueName: \"kubernetes.io/projected/c9acd468-3d02-4521-9db7-33e9dff0259f-kube-api-access-vcvhb\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.044612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.044500    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.044612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.044515    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vbgb\" (UniqueName: \"kubernetes.io/projected/3621a9d6-ea49-415f-a6a9-5ce99245d111-kube-api-access-4vbgb\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.044612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.044537    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3621a9d6-ea49-415f-a6a9-5ce99245d111-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.044612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.044550    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9acd468-3d02-4521-9db7-33e9dff0259f-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.047396 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.047335    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-kube-api-access-pnvh6" pod "ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" (UID: "ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3"). InnerVolumeSpecName "kube-api-access-pnvh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:24.129824 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.129728    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs" event={"ID":"c9acd468-3d02-4521-9db7-33e9dff0259f","Type":"ContainerDied","Data":"9f0fd6ad2263913994c11e943071e1d6d7b5e48a4d91cb41c6d8910ada16d79a"}
Oct 30 05:00:24.130631 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.130609    5072 scope.go:122] "RemoveContainer" containerID="1c263d516a23f58da2df6c566c703f2c6262970fd440850dc555ad7719d26e4c"
Oct 30 05:00:24.130858 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.129747    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"
Oct 30 05:00:24.147778 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.147373    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnvh6\" (UniqueName: \"kubernetes.io/projected/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3-kube-api-access-pnvh6\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.152150 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.151304    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt" event={"ID":"ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3","Type":"ContainerDied","Data":"b3eea4e83e34a7a83019942e85f085dc25781e03a08e5ead06ec1722622d1f62"}
Oct 30 05:00:24.152150 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.151400    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"
Oct 30 05:00:24.238454 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.238429    5072 scope.go:122] "RemoveContainer" containerID="3f083f53651eaeddcc573eac3550c79298c2eae1e97902fd09498c09da9fb972"
Oct 30 05:00:24.252158 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.251298    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n" event={"ID":"3621a9d6-ea49-415f-a6a9-5ce99245d111","Type":"ContainerDied","Data":"686f7959bda36c40185c3272dfa8cf20ada2bc1e9bb4ae66ec5c6a28ea31441c"}
Oct 30 05:00:24.252158 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.251416    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"
Oct 30 05:00:24.356618 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.356586    5072 scope.go:122] "RemoveContainer" containerID="b17b33c41e840b88e72414cbd8f9addc8f653774814c1210d74b4c509a0b6fe6"
Oct 30 05:00:24.378786 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.378429    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2" event={"ID":"fc366c3b-6526-4575-899f-df191a5971de","Type":"ContainerDied","Data":"6fc771849ccab5fc3684ae8b2ee962160938999a54111927e20717d3c12fe9e8"}
Oct 30 05:00:24.378786 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.378565    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"
Oct 30 05:00:24.430841 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.430809    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379" path="/var/lib/kubelet/pods/06e8c6d6-1834-4ac0-a21d-998da69a4379/volumes"
Oct 30 05:00:24.431752 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.431731    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" path="/var/lib/kubelet/pods/0e846f7c-186a-4e49-ba15-6ff6c564bc13/volumes"
Oct 30 05:00:24.432518 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.432499    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212e51ce-0a69-4809-822f-6f6fe6bf8773" path="/var/lib/kubelet/pods/212e51ce-0a69-4809-822f-6f6fe6bf8773/volumes"
Oct 30 05:00:24.433625 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.433607    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" path="/var/lib/kubelet/pods/6fa98861-f2c2-47aa-bb83-54ddf8c5c715/volumes"
Oct 30 05:00:24.434321 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.434305    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d" path="/var/lib/kubelet/pods/7a540e59-8239-423a-ac80-aa84d7b4821d/volumes"
Oct 30 05:00:24.434980 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.434963    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" path="/var/lib/kubelet/pods/7fdb04a5-7116-41a3-80cd-f1169b75cc2b/volumes"
Oct 30 05:00:24.436127 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.436049    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" path="/var/lib/kubelet/pods/8dff5e9e-5890-4180-8c30-ee2b8a1b4e61/volumes"
Oct 30 05:00:24.436751 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.436734    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"]
Oct 30 05:00:24.436888 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.436877    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888940-6txwt"]
Oct 30 05:00:24.442892 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.441787    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"
Oct 30 05:00:24.463132 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.459030    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52b50b1-dfe7-4316-9918-61d44366cd60-kube-api-access-mqz4x\" (UniqueName: \"kubernetes.io/projected/f52b50b1-dfe7-4316-9918-61d44366cd60-kube-api-access-mqz4x\") pod \"f52b50b1-dfe7-4316-9918-61d44366cd60\" (UID: \"f52b50b1-dfe7-4316-9918-61d44366cd60\") "
Oct 30 05:00:24.463132 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.459386    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f52b50b1-dfe7-4316-9918-61d44366cd60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52b50b1-dfe7-4316-9918-61d44366cd60-maas-api-service-ca\") pod \"f52b50b1-dfe7-4316-9918-61d44366cd60\" (UID: \"f52b50b1-dfe7-4316-9918-61d44366cd60\") "
Oct 30 05:00:24.463132 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.461579    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52b50b1-dfe7-4316-9918-61d44366cd60-maas-api-service-ca" pod "f52b50b1-dfe7-4316-9918-61d44366cd60" (UID: "f52b50b1-dfe7-4316-9918-61d44366cd60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:24.466847 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.466819    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52b50b1-dfe7-4316-9918-61d44366cd60-kube-api-access-mqz4x" pod "f52b50b1-dfe7-4316-9918-61d44366cd60" (UID: "f52b50b1-dfe7-4316-9918-61d44366cd60"). InnerVolumeSpecName "kube-api-access-mqz4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:24.476055 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.476030    5072 scope.go:122] "RemoveContainer" containerID="eb59b27c558ff05dcb8e0d5428ec00d325b83f0f65b6a7471ed0d907f3b264ae"
Oct 30 05:00:24.537462 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.534301    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"
Oct 30 05:00:24.562620 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.561177    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ca5f62-df90-42b8-91cb-dadacd5b3668-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ca5f62-df90-42b8-91cb-dadacd5b3668-maas-api-service-ca\") pod \"82ca5f62-df90-42b8-91cb-dadacd5b3668\" (UID: \"82ca5f62-df90-42b8-91cb-dadacd5b3668\") "
Oct 30 05:00:24.562620 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.561357    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ca5f62-df90-42b8-91cb-dadacd5b3668-kube-api-access-j8n6b\" (UniqueName: \"kubernetes.io/projected/82ca5f62-df90-42b8-91cb-dadacd5b3668-kube-api-access-j8n6b\") pod \"82ca5f62-df90-42b8-91cb-dadacd5b3668\" (UID: \"82ca5f62-df90-42b8-91cb-dadacd5b3668\") "
Oct 30 05:00:24.562620 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.562583    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ca5f62-df90-42b8-91cb-dadacd5b3668-maas-api-service-ca" pod "82ca5f62-df90-42b8-91cb-dadacd5b3668" (UID: "82ca5f62-df90-42b8-91cb-dadacd5b3668"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:24.563563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.563190    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52b50b1-dfe7-4316-9918-61d44366cd60-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.563563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.563214    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ca5f62-df90-42b8-91cb-dadacd5b3668-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.563563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.563227    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqz4x\" (UniqueName: \"kubernetes.io/projected/f52b50b1-dfe7-4316-9918-61d44366cd60-kube-api-access-mqz4x\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.581437 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.581390    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ca5f62-df90-42b8-91cb-dadacd5b3668-kube-api-access-j8n6b" pod "82ca5f62-df90-42b8-91cb-dadacd5b3668" (UID: "82ca5f62-df90-42b8-91cb-dadacd5b3668"). InnerVolumeSpecName "kube-api-access-j8n6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:24.658293 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.658226    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"
Oct 30 05:00:24.664796 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.664763    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8n6b\" (UniqueName: \"kubernetes.io/projected/82ca5f62-df90-42b8-91cb-dadacd5b3668-kube-api-access-j8n6b\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.666994 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.666964    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"
Oct 30 05:00:24.677292 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.677267    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"
Oct 30 05:00:24.731405 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.731370    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"
Oct 30 05:00:24.766770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.766327    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e53d6987-db5e-468a-9afe-2c0603cbf408-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53d6987-db5e-468a-9afe-2c0603cbf408-maas-api-service-ca\") pod \"e53d6987-db5e-468a-9afe-2c0603cbf408\" (UID: \"e53d6987-db5e-468a-9afe-2c0603cbf408\") "
Oct 30 05:00:24.766770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.766398    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927eaf22-dd8b-4bef-a48c-fff361ead463-kube-api-access-qksl6\" (UniqueName: \"kubernetes.io/projected/927eaf22-dd8b-4bef-a48c-fff361ead463-kube-api-access-qksl6\") pod \"927eaf22-dd8b-4bef-a48c-fff361ead463\" (UID: \"927eaf22-dd8b-4bef-a48c-fff361ead463\") "
Oct 30 05:00:24.766770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.766447    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a82ff41-ac42-48bc-81c4-24e539017282-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a82ff41-ac42-48bc-81c4-24e539017282-maas-api-service-ca\") pod \"2a82ff41-ac42-48bc-81c4-24e539017282\" (UID: \"2a82ff41-ac42-48bc-81c4-24e539017282\") "
Oct 30 05:00:24.766770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.766528    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5cb00e5-a10c-4e5a-846b-7e10e452611e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5cb00e5-a10c-4e5a-846b-7e10e452611e-maas-api-service-ca\") pod \"a5cb00e5-a10c-4e5a-846b-7e10e452611e\" (UID: \"a5cb00e5-a10c-4e5a-846b-7e10e452611e\") "
Oct 30 05:00:24.768024 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.767991    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e53d6987-db5e-468a-9afe-2c0603cbf408-kube-api-access-sqx7p\" (UniqueName: \"kubernetes.io/projected/e53d6987-db5e-468a-9afe-2c0603cbf408-kube-api-access-sqx7p\") pod \"e53d6987-db5e-468a-9afe-2c0603cbf408\" (UID: \"e53d6987-db5e-468a-9afe-2c0603cbf408\") "
Oct 30 05:00:24.768185 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.768054    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5cb00e5-a10c-4e5a-846b-7e10e452611e-kube-api-access-qv874\" (UniqueName: \"kubernetes.io/projected/a5cb00e5-a10c-4e5a-846b-7e10e452611e-kube-api-access-qv874\") pod \"a5cb00e5-a10c-4e5a-846b-7e10e452611e\" (UID: \"a5cb00e5-a10c-4e5a-846b-7e10e452611e\") "
Oct 30 05:00:24.768185 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.768126    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a82ff41-ac42-48bc-81c4-24e539017282-kube-api-access-84nlf\" (UniqueName: \"kubernetes.io/projected/2a82ff41-ac42-48bc-81c4-24e539017282-kube-api-access-84nlf\") pod \"2a82ff41-ac42-48bc-81c4-24e539017282\" (UID: \"2a82ff41-ac42-48bc-81c4-24e539017282\") "
Oct 30 05:00:24.768297 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.768262    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/927eaf22-dd8b-4bef-a48c-fff361ead463-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/927eaf22-dd8b-4bef-a48c-fff361ead463-maas-api-service-ca\") pod \"927eaf22-dd8b-4bef-a48c-fff361ead463\" (UID: \"927eaf22-dd8b-4bef-a48c-fff361ead463\") "
Oct 30 05:00:24.769437 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.769410    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927eaf22-dd8b-4bef-a48c-fff361ead463-maas-api-service-ca" pod "927eaf22-dd8b-4bef-a48c-fff361ead463" (UID: "927eaf22-dd8b-4bef-a48c-fff361ead463"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:24.769737 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.769720    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a82ff41-ac42-48bc-81c4-24e539017282-maas-api-service-ca" pod "2a82ff41-ac42-48bc-81c4-24e539017282" (UID: "2a82ff41-ac42-48bc-81c4-24e539017282"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:24.770012 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.769995    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5cb00e5-a10c-4e5a-846b-7e10e452611e-maas-api-service-ca" pod "a5cb00e5-a10c-4e5a-846b-7e10e452611e" (UID: "a5cb00e5-a10c-4e5a-846b-7e10e452611e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:24.775537 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.774577    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53d6987-db5e-468a-9afe-2c0603cbf408-maas-api-service-ca" pod "e53d6987-db5e-468a-9afe-2c0603cbf408" (UID: "e53d6987-db5e-468a-9afe-2c0603cbf408"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:24.775537 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.774630    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"]
Oct 30 05:00:24.775880 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.775853    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53d6987-db5e-468a-9afe-2c0603cbf408-kube-api-access-sqx7p" pod "e53d6987-db5e-468a-9afe-2c0603cbf408" (UID: "e53d6987-db5e-468a-9afe-2c0603cbf408"). InnerVolumeSpecName "kube-api-access-sqx7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:24.776706 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.776683    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927eaf22-dd8b-4bef-a48c-fff361ead463-kube-api-access-qksl6" pod "927eaf22-dd8b-4bef-a48c-fff361ead463" (UID: "927eaf22-dd8b-4bef-a48c-fff361ead463"). InnerVolumeSpecName "kube-api-access-qksl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:24.777358 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.777335    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cb00e5-a10c-4e5a-846b-7e10e452611e-kube-api-access-qv874" pod "a5cb00e5-a10c-4e5a-846b-7e10e452611e" (UID: "a5cb00e5-a10c-4e5a-846b-7e10e452611e"). InnerVolumeSpecName "kube-api-access-qv874". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:24.779794 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.779766    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-503804-29888940-28pfs"]
Oct 30 05:00:24.780385 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.780362    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a82ff41-ac42-48bc-81c4-24e539017282-kube-api-access-84nlf" pod "2a82ff41-ac42-48bc-81c4-24e539017282" (UID: "2a82ff41-ac42-48bc-81c4-24e539017282"). InnerVolumeSpecName "kube-api-access-84nlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:24.872442 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.872392    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqx7p\" (UniqueName: \"kubernetes.io/projected/e53d6987-db5e-468a-9afe-2c0603cbf408-kube-api-access-sqx7p\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.872442 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.872439    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv874\" (UniqueName: \"kubernetes.io/projected/a5cb00e5-a10c-4e5a-846b-7e10e452611e-kube-api-access-qv874\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.873012 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.872456    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84nlf\" (UniqueName: \"kubernetes.io/projected/2a82ff41-ac42-48bc-81c4-24e539017282-kube-api-access-84nlf\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.873012 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.872470    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/927eaf22-dd8b-4bef-a48c-fff361ead463-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.873012 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.872483    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53d6987-db5e-468a-9afe-2c0603cbf408-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.873012 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.872497    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qksl6\" (UniqueName: \"kubernetes.io/projected/927eaf22-dd8b-4bef-a48c-fff361ead463-kube-api-access-qksl6\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.873012 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.872510    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a82ff41-ac42-48bc-81c4-24e539017282-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.873012 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.872524    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5cb00e5-a10c-4e5a-846b-7e10e452611e-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.936125 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.935982    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"]
Oct 30 05:00:24.943877 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.943844    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82bec5-29888940-f2h5n"]
Oct 30 05:00:24.973124 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.972181    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"]
Oct 30 05:00:24.977542 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:24.977521    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bbce72-29888940-vh8s2"]
Oct 30 05:00:25.403676 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.403597    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"
Oct 30 05:00:25.403676 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.403627    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf" event={"ID":"2a82ff41-ac42-48bc-81c4-24e539017282","Type":"ContainerDied","Data":"56ea82144e74659d1b4657ae9478cf31c65b71a76f52677d364121081f571fa1"}
Oct 30 05:00:25.403910 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.403688    5072 scope.go:122] "RemoveContainer" containerID="30ca9bd881938f1a754802dc49a62fcd0bdda8fc1ab0f6f27ecfb665f57e9927"
Oct 30 05:00:25.405712 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.405687    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"
Oct 30 05:00:25.406069 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.405688    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5" event={"ID":"e53d6987-db5e-468a-9afe-2c0603cbf408","Type":"ContainerDied","Data":"279813d32e6685eb916520774045eeed5079cfd5511059b430ff1f7a23502b72"}
Oct 30 05:00:25.407430 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.407405    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m" event={"ID":"927eaf22-dd8b-4bef-a48c-fff361ead463","Type":"ContainerDied","Data":"997d984231489801a631803e20c170d6d021780511401aae7da3679fc5d5ca12"}
Oct 30 05:00:25.407601 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.407582    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"
Oct 30 05:00:25.413228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.413201    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7" event={"ID":"82ca5f62-df90-42b8-91cb-dadacd5b3668","Type":"ContainerDied","Data":"9d20624bc36d147d343ea69c5a081fa0116887d95b85c6c40e6103d6066fa39c"}
Oct 30 05:00:25.413323 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.413211    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"
Oct 30 05:00:25.415453 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.415431    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d" event={"ID":"a5cb00e5-a10c-4e5a-846b-7e10e452611e","Type":"ContainerDied","Data":"4d9d04323067c7a73d13a2dc9d6076e5c5ff44becc42fed97ef596cb6a69679b"}
Oct 30 05:00:25.415646 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.415628    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"
Oct 30 05:00:25.418564 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.418540    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv" event={"ID":"f52b50b1-dfe7-4316-9918-61d44366cd60","Type":"ContainerDied","Data":"818d70685dc12f54d10f0a41924ce014863b0a5cde881dd6a2385e1f98aa665c"}
Oct 30 05:00:25.418674 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.418631    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"
Oct 30 05:00:25.420359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.420325    5072 scope.go:122] "RemoveContainer" containerID="915c47259b17c9c3c2e88d429c38ab1878306612eee9475eeb617d784e291376"
Oct 30 05:00:25.435743 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.435716    5072 scope.go:122] "RemoveContainer" containerID="75003dc0e4ca30df3d3d792e62520746a3c18c8399a45154150d47a6536655f4"
Oct 30 05:00:25.446316 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.446294    5072 scope.go:122] "RemoveContainer" containerID="76d06d6483ef4c9a04196977108dfac33a8d8339dd82a957ceac4c54a195e03c"
Oct 30 05:00:25.457220 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.457196    5072 scope.go:122] "RemoveContainer" containerID="4bb0331ab460c124dba98b9149172138bd1e611d49563b9669d085b95662303f"
Oct 30 05:00:25.470839 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.470813    5072 scope.go:122] "RemoveContainer" containerID="5b5ad3e911c8cd7800b690d3405bf8ade1bd4665026717ed81f72da8617e363a"
Oct 30 05:00:25.538119 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.538062    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"]
Oct 30 05:00:25.542388 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.542366    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888940-r7zsf"]
Oct 30 05:00:25.576705 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.576673    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"]
Oct 30 05:00:25.585644 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.585611    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91705f-29888940-dqzqv"]
Oct 30 05:00:25.620202 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.620171    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"]
Oct 30 05:00:25.628480 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.628453    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b195d0-29888940-v966m"]
Oct 30 05:00:25.666848 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.666777    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"]
Oct 30 05:00:25.671016 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.670994    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0d6e7-29888940-vmz5d"]
Oct 30 05:00:25.706712 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.706678    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"]
Oct 30 05:00:25.710777 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.710754    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888940-ng8p7"]
Oct 30 05:00:25.744727 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.744695    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"]
Oct 30 05:00:25.748630 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:25.748609    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5f1508-29888940-kg9f5"]
Oct 30 05:00:26.412923 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:26.412892    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a82ff41-ac42-48bc-81c4-24e539017282" path="/var/lib/kubelet/pods/2a82ff41-ac42-48bc-81c4-24e539017282/volumes"
Oct 30 05:00:26.413420 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:26.413348    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111" path="/var/lib/kubelet/pods/3621a9d6-ea49-415f-a6a9-5ce99245d111/volumes"
Oct 30 05:00:26.413742 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:26.413726    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668" path="/var/lib/kubelet/pods/82ca5f62-df90-42b8-91cb-dadacd5b3668/volumes"
Oct 30 05:00:26.414428 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:26.414412    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463" path="/var/lib/kubelet/pods/927eaf22-dd8b-4bef-a48c-fff361ead463/volumes"
Oct 30 05:00:26.414827 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:26.414810    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" path="/var/lib/kubelet/pods/a5cb00e5-a10c-4e5a-846b-7e10e452611e/volumes"
Oct 30 05:00:26.415224 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:26.415210    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9acd468-3d02-4521-9db7-33e9dff0259f" path="/var/lib/kubelet/pods/c9acd468-3d02-4521-9db7-33e9dff0259f/volumes"
Oct 30 05:00:26.415856 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:26.415841    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408" path="/var/lib/kubelet/pods/e53d6987-db5e-468a-9afe-2c0603cbf408/volumes"
Oct 30 05:00:26.416244 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:26.416230    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" path="/var/lib/kubelet/pods/ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3/volumes"
Oct 30 05:00:26.416609 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:26.416597    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60" path="/var/lib/kubelet/pods/f52b50b1-dfe7-4316-9918-61d44366cd60/volumes"
Oct 30 05:00:26.417279 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:26.417264    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc366c3b-6526-4575-899f-df191a5971de" path="/var/lib/kubelet/pods/fc366c3b-6526-4575-899f-df191a5971de/volumes"
Oct 30 05:00:41.753698 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:00:41.753615    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:00:46.649473 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.649431    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"]
Oct 30 05:00:46.650462 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650441    5072 cpu_manager.go:414] "containerMap: removing container" podUID="fc366c3b-6526-4575-899f-df191a5971de" containerName="cleanup"
Oct 30 05:00:46.650462 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650461    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc366c3b-6526-4575-899f-df191a5971de" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650472    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650478    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650500    5072 cpu_manager.go:414] "containerMap: removing container" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650505    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650515    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650519    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650527    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650531    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650538    5072 cpu_manager.go:414] "containerMap: removing container" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650542    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650548    5072 cpu_manager.go:414] "containerMap: removing container" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650552    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650558    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650561    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650569    5072 cpu_manager.go:414] "containerMap: removing container" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650574    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650586    5072 cpu_manager.go:414] "containerMap: removing container" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650593    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650606    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650610    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650617    5072 cpu_manager.go:414] "containerMap: removing container" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650621    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650629    5072 cpu_manager.go:414] "containerMap: removing container" podUID="c9acd468-3d02-4521-9db7-33e9dff0259f" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650633    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9acd468-3d02-4521-9db7-33e9dff0259f" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650639    5072 cpu_manager.go:414] "containerMap: removing container" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5" containerName="cleanup"
Oct 30 05:00:46.650634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650643    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650658    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650662    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650671    5072 cpu_manager.go:414] "containerMap: removing container" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650676    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650685    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650690    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650696    5072 cpu_manager.go:414] "containerMap: removing container" podUID="fc366c3b-6526-4575-899f-df191a5971de" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650700    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc366c3b-6526-4575-899f-df191a5971de" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650706    5072 cpu_manager.go:414] "containerMap: removing container" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650710    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650720    5072 cpu_manager.go:414] "containerMap: removing container" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650723    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650729    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2a82ff41-ac42-48bc-81c4-24e539017282" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650733    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a82ff41-ac42-48bc-81c4-24e539017282" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650739    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2a82ff41-ac42-48bc-81c4-24e539017282" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650742    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a82ff41-ac42-48bc-81c4-24e539017282" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650751    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650755    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650761    5072 cpu_manager.go:414] "containerMap: removing container" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650765    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650772    5072 cpu_manager.go:414] "containerMap: removing container" podUID="212e51ce-0a69-4809-822f-6f6fe6bf8773" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650776    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212e51ce-0a69-4809-822f-6f6fe6bf8773" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650783    5072 cpu_manager.go:414] "containerMap: removing container" podUID="212e51ce-0a69-4809-822f-6f6fe6bf8773" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650787    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212e51ce-0a69-4809-822f-6f6fe6bf8773" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650793    5072 cpu_manager.go:414] "containerMap: removing container" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650797    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650803    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650807    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650812    5072 cpu_manager.go:414] "containerMap: removing container" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650818    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650827    5072 cpu_manager.go:414] "containerMap: removing container" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650831    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650838    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650843    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650849    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650854    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650859    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650864    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650870    5072 cpu_manager.go:414] "containerMap: removing container" podUID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650873    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650884    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650889    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650895    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650902    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650911    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650917    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650926    5072 cpu_manager.go:414] "containerMap: removing container" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650932    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650942    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650948    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650961    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650971    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650983    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.650991    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651007    5072 cpu_manager.go:414] "containerMap: removing container" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651015    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651026    5072 cpu_manager.go:414] "containerMap: removing container" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651032    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651042    5072 cpu_manager.go:414] "containerMap: removing container" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651049    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651062    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651071    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651103    5072 cpu_manager.go:414] "containerMap: removing container" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651110    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651123    5072 cpu_manager.go:414] "containerMap: removing container" podUID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651128    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651138    5072 cpu_manager.go:414] "containerMap: removing container" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651143    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651150    5072 cpu_manager.go:414] "containerMap: removing container" podUID="c9acd468-3d02-4521-9db7-33e9dff0259f" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651154    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9acd468-3d02-4521-9db7-33e9dff0259f" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651163    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651167    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651173    5072 cpu_manager.go:414] "containerMap: removing container" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651177    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651186    5072 cpu_manager.go:414] "containerMap: removing container" podUID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651190    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651196    5072 cpu_manager.go:414] "containerMap: removing container" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651200    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651364    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651372    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651379    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651385    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651391    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651396    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651403    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651409    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651416    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651420    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651425    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc366c3b-6526-4575-899f-df191a5971de" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651430    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651435    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651440    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651445    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651450    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651455    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a82ff41-ac42-48bc-81c4-24e539017282" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651461    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651468    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="212e51ce-0a69-4809-822f-6f6fe6bf8773" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651472    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651478    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651482    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651486    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651494    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9acd468-3d02-4521-9db7-33e9dff0259f" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651499    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651504    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651509    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651514    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651520    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc366c3b-6526-4575-899f-df191a5971de" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651527    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651540    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651550    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651557    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651563    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651569    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651574    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651578    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651584    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651590    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651595    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9acd468-3d02-4521-9db7-33e9dff0259f" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651601    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651607    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651614    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9acd468-3d02-4521-9db7-33e9dff0259f" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651620    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651627    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651632    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3621a9d6-ea49-415f-a6a9-5ce99245d111" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651636    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a82ff41-ac42-48bc-81c4-24e539017282" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651643    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a82ff41-ac42-48bc-81c4-24e539017282" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651650    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="212e51ce-0a69-4809-822f-6f6fe6bf8773" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651654    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651659    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dff5e9e-5890-4180-8c30-ee2b8a1b4e61" containerName="cleanup"
Oct 30 05:00:46.652059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.651667    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" containerName="cleanup"
Oct 30 05:00:46.658728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.656984    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.677634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.677608    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"]
Oct 30 05:00:46.684122 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.682190    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e592d7\"/\"openshift-service-ca.crt\""
Oct 30 05:00:46.684122 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.682313    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e592d7\"/\"kube-root-ca.crt\""
Oct 30 05:00:46.692141 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.688283    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e592d7\"/\"test-model-e592d7-kserve-self-signed-certs\""
Oct 30 05:00:46.692141 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.688394    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e592d7\"/\"default-dockercfg-hvhlw\""
Oct 30 05:00:46.692563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.692539    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tmp-dir\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.692767 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.692745    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxz48\" (UniqueName: \"kubernetes.io/projected/0c4a4c41-7fff-4875-9de2-ee671ed2a976-kube-api-access-wxz48\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.692830 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.692815    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-home\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.692868 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.692855    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-dshm\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.692983 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.692966    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tls-certs\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.693054 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.693039    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-model-cache\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.794164 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.794124    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tmp-dir\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.794382 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.794253    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxz48\" (UniqueName: \"kubernetes.io/projected/0c4a4c41-7fff-4875-9de2-ee671ed2a976-kube-api-access-wxz48\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.794382 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.794309    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-home\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.794382 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.794340    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-dshm\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.794572 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.794436    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tls-certs\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.794572 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.794507    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-model-cache\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.794674 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.794576    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tmp-dir\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.795231 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.794897    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-model-cache\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.795231 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.794960    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-home\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.797129 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.797064    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-dshm\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.797504 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.797483    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tls-certs\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.818467 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.818436    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxz48\" (UniqueName: \"kubernetes.io/projected/0c4a4c41-7fff-4875-9de2-ee671ed2a976-kube-api-access-wxz48\") pod \"test-model-e592d7-kserve-7c8d5597d7-sns2h\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") " pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:46.983100 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:46.982160    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:47.322042 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:47.322009    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c4a4c41_7fff_4875_9de2_ee671ed2a976.slice/crio-b08ad1bd37656a96d6f3bc296fb873de71fe5d2a6ad36d85149826f7ec1655e5 WatchSource:0}: Error finding container b08ad1bd37656a96d6f3bc296fb873de71fe5d2a6ad36d85149826f7ec1655e5: Status 404 returned error can't find the container with id b08ad1bd37656a96d6f3bc296fb873de71fe5d2a6ad36d85149826f7ec1655e5
Oct 30 05:00:47.322556 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:47.322533    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"]
Oct 30 05:00:47.646919 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:47.646823    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h" event={"ID":"0c4a4c41-7fff-4875-9de2-ee671ed2a976","Type":"ContainerStarted","Data":"ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153"}
Oct 30 05:00:47.646919 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:47.646875    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h" event={"ID":"0c4a4c41-7fff-4875-9de2-ee671ed2a976","Type":"ContainerStarted","Data":"b08ad1bd37656a96d6f3bc296fb873de71fe5d2a6ad36d85149826f7ec1655e5"}
Oct 30 05:00:47.692553 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:47.692493    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h" podStartSLOduration=1.692473327 podStartE2EDuration="1.692473327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:47.689804114 +0000 UTC m=+3827.431227214" watchObservedRunningTime="2026-10-30 05:00:47.692473327 +0000 UTC m=+3827.433896429"
Oct 30 05:00:47.983034 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:47.983006    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:54.060412 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:54.053145    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"]
Oct 30 05:00:54.077937 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:54.077898    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0421600d-dn5tg"]
Oct 30 05:00:54.411547 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:54.411448    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dddc8322-0daf-4ea0-bce8-9fba2f71e0e2" path="/var/lib/kubelet/pods/dddc8322-0daf-4ea0-bce8-9fba2f71e0e2/volumes"
Oct 30 05:00:56.982670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:56.982635    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:56.996013 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:56.995979    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:57.200778 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.200746    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"]
Oct 30 05:00:57.201650 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201634    5072 cpu_manager.go:414] "containerMap: removing container" podUID="c9acd468-3d02-4521-9db7-33e9dff0259f" containerName="cleanup"
Oct 30 05:00:57.201704 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201652    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9acd468-3d02-4521-9db7-33e9dff0259f" containerName="cleanup"
Oct 30 05:00:57.201704 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201667    5072 cpu_manager.go:414] "containerMap: removing container" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5" containerName="cleanup"
Oct 30 05:00:57.201704 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201673    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5" containerName="cleanup"
Oct 30 05:00:57.201704 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201689    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d" containerName="cleanup"
Oct 30 05:00:57.201704 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201694    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a540e59-8239-423a-ac80-aa84d7b4821d" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201707    5072 cpu_manager.go:414] "containerMap: removing container" podUID="fc366c3b-6526-4575-899f-df191a5971de" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201712    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc366c3b-6526-4575-899f-df191a5971de" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201721    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2a82ff41-ac42-48bc-81c4-24e539017282" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201726    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a82ff41-ac42-48bc-81c4-24e539017282" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201736    5072 cpu_manager.go:414] "containerMap: removing container" podUID="212e51ce-0a69-4809-822f-6f6fe6bf8773" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201741    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212e51ce-0a69-4809-822f-6f6fe6bf8773" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201758    5072 cpu_manager.go:414] "containerMap: removing container" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201765    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201784    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201791    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201801    5072 cpu_manager.go:414] "containerMap: removing container" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201806    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201813    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201818    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52b50b1-dfe7-4316-9918-61d44366cd60" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201853    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201858    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201864    5072 cpu_manager.go:414] "containerMap: removing container" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201868    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8802708-e588-476f-b55b-bce1a3efa0cd" containerName="cleanup"
Oct 30 05:00:57.201866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201876    5072 cpu_manager.go:414] "containerMap: removing container" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.201880    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ca5f62-df90-42b8-91cb-dadacd5b3668" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202010    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="212e51ce-0a69-4809-822f-6f6fe6bf8773" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202018    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5cb00e5-a10c-4e5a-846b-7e10e452611e" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202025    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdb04a5-7116-41a3-80cd-f1169b75cc2b" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202032    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f638f03d-08bb-4937-aa89-2198ac4a4c02" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202039    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="804abcc6-7570-4f94-be22-d2c8b12f2108" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202047    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e846f7c-186a-4e49-ba15-6ff6c564bc13" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202052    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa98861-f2c2-47aa-bb83-54ddf8c5c715" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202057    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="927eaf22-dd8b-4bef-a48c-fff361ead463" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202064    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc366c3b-6526-4575-899f-df191a5971de" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202071    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6f868c-5bea-49e5-a136-ab5862d1841d" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202095    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e8c6d6-1834-4ac0-a21d-998da69a4379" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202099    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7fc8fe-4ea2-4e79-9e10-fd0994f364e3" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202105    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e82ab9-06b8-4c71-b46a-76348dee31b5" containerName="cleanup"
Oct 30 05:00:57.202519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.202115    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53d6987-db5e-468a-9afe-2c0603cbf408" containerName="cleanup"
Oct 30 05:00:57.207452 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.207433    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:00:57.223499 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.223471    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:00:57.223820 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.223794    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:00:57.224434 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.224388    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"]
Oct 30 05:00:57.341178 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.341091    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7dc8b74-26jtf\" (UID: \"a5c7ea18-975b-4f96-80aa-a4f20358d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:00:57.341178 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.341159    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7dc8b74-26jtf\" (UID: \"a5c7ea18-975b-4f96-80aa-a4f20358d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:00:57.341425 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.341273    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwmkg\" (UniqueName: \"kubernetes.io/projected/a5c7ea18-975b-4f96-80aa-a4f20358d963-kube-api-access-bwmkg\") pod \"maas-api-revoke-sub-e7dc8b74-26jtf\" (UID: \"a5c7ea18-975b-4f96-80aa-a4f20358d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:00:57.442950 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.442917    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7dc8b74-26jtf\" (UID: \"a5c7ea18-975b-4f96-80aa-a4f20358d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:00:57.443226 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.442962    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7dc8b74-26jtf\" (UID: \"a5c7ea18-975b-4f96-80aa-a4f20358d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:00:57.443226 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.443029    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwmkg\" (UniqueName: \"kubernetes.io/projected/a5c7ea18-975b-4f96-80aa-a4f20358d963-kube-api-access-bwmkg\") pod \"maas-api-revoke-sub-e7dc8b74-26jtf\" (UID: \"a5c7ea18-975b-4f96-80aa-a4f20358d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:00:57.443748 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.443724    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7dc8b74-26jtf\" (UID: \"a5c7ea18-975b-4f96-80aa-a4f20358d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:00:57.445529 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.445508    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7dc8b74-26jtf\" (UID: \"a5c7ea18-975b-4f96-80aa-a4f20358d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:00:57.467602 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.467505    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwmkg\" (UniqueName: \"kubernetes.io/projected/a5c7ea18-975b-4f96-80aa-a4f20358d963-kube-api-access-bwmkg\") pod \"maas-api-revoke-sub-e7dc8b74-26jtf\" (UID: \"a5c7ea18-975b-4f96-80aa-a4f20358d963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:00:57.526677 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.526404    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:00:57.784135 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.784070    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:00:57.934677 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.934647    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"]
Oct 30 05:00:57.946943 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.946909    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:00:57.949241 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.949189    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"]
Oct 30 05:00:57.949558 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:57.949531    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c7ea18_975b_4f96_80aa_a4f20358d963.slice/crio-faff434da252aafe33a0154e906cfea1ed1589e309c1cb94e5d929e0889aadd9 WatchSource:0}: Error finding container faff434da252aafe33a0154e906cfea1ed1589e309c1cb94e5d929e0889aadd9: Status 404 returned error can't find the container with id faff434da252aafe33a0154e906cfea1ed1589e309c1cb94e5d929e0889aadd9
Oct 30 05:00:57.958820 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:57.958796    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"]
Oct 30 05:00:58.086725 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.086689    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d542375d-nhd2n\" (UID: \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:00:58.087151 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.086934    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d542375d-nhd2n\" (UID: \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:00:58.087151 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.087000    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhp9d\" (UniqueName: \"kubernetes.io/projected/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-kube-api-access-qhp9d\") pod \"maas-api-revoke-sub-d542375d-nhd2n\" (UID: \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:00:58.199098 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.196643    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d542375d-nhd2n\" (UID: \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:00:58.199098 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.196787    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhp9d\" (UniqueName: \"kubernetes.io/projected/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-kube-api-access-qhp9d\") pod \"maas-api-revoke-sub-d542375d-nhd2n\" (UID: \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:00:58.199098 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.196922    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d542375d-nhd2n\" (UID: \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:00:58.199098 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.198008    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d542375d-nhd2n\" (UID: \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:00:58.242723 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.242640    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d542375d-nhd2n\" (UID: \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:00:58.266101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.258419    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhp9d\" (UniqueName: \"kubernetes.io/projected/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-kube-api-access-qhp9d\") pod \"maas-api-revoke-sub-d542375d-nhd2n\" (UID: \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:00:58.297658 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.297435    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:00:58.664651 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.664616    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"]
Oct 30 05:00:58.671403 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:00:58.671366    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode595a127_8671_4ba9_a20f_7fbe5d92f3c6.slice/crio-1576e27d16d3e7a40b437d8cedb701430d9b7966f6762ec12140b4a8d0681247 WatchSource:0}: Error finding container 1576e27d16d3e7a40b437d8cedb701430d9b7966f6762ec12140b4a8d0681247: Status 404 returned error can't find the container with id 1576e27d16d3e7a40b437d8cedb701430d9b7966f6762ec12140b4a8d0681247
Oct 30 05:00:58.786564 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.786516    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n" event={"ID":"e595a127-8671-4ba9-a20f-7fbe5d92f3c6","Type":"ContainerStarted","Data":"1576e27d16d3e7a40b437d8cedb701430d9b7966f6762ec12140b4a8d0681247"}
Oct 30 05:00:58.793505 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.793473    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf" event={"ID":"a5c7ea18-975b-4f96-80aa-a4f20358d963","Type":"ContainerStarted","Data":"804cbdd3632ea7518375c56f15951e639f1252de907a7a6aed06f35ef3560d2c"}
Oct 30 05:00:58.793660 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.793524    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf" event={"ID":"a5c7ea18-975b-4f96-80aa-a4f20358d963","Type":"ContainerStarted","Data":"faff434da252aafe33a0154e906cfea1ed1589e309c1cb94e5d929e0889aadd9"}
Oct 30 05:00:58.835739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:58.835693    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf" podStartSLOduration=1.8356810449999998 podStartE2EDuration="1.835681045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:58.834588791 +0000 UTC m=+3838.576011896" watchObservedRunningTime="2026-10-30 05:00:58.835681045 +0000 UTC m=+3838.577104142"
Oct 30 05:00:59.804434 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:59.804393    5072 generic.go:356] "Generic (PLEG): container finished" podID="e595a127-8671-4ba9-a20f-7fbe5d92f3c6" containerID="bcc4b50be5f3aae4dbc464e55bfee70cce5268b2fe09f70572176181fc8ecaa2" exitCode=0
Oct 30 05:00:59.804882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:59.804513    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n" event={"ID":"e595a127-8671-4ba9-a20f-7fbe5d92f3c6","Type":"ContainerDied","Data":"bcc4b50be5f3aae4dbc464e55bfee70cce5268b2fe09f70572176181fc8ecaa2"}
Oct 30 05:00:59.806451 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:59.806423    5072 generic.go:356] "Generic (PLEG): container finished" podID="a5c7ea18-975b-4f96-80aa-a4f20358d963" containerID="804cbdd3632ea7518375c56f15951e639f1252de907a7a6aed06f35ef3560d2c" exitCode=0
Oct 30 05:00:59.806597 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:00:59.806454    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf" event={"ID":"a5c7ea18-975b-4f96-80aa-a4f20358d963","Type":"ContainerDied","Data":"804cbdd3632ea7518375c56f15951e639f1252de907a7a6aed06f35ef3560d2c"}
Oct 30 05:01:01.039702 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.039671    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:01:01.043175 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.043153    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:01:01.167301 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.167213    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-xks-ca\") pod \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\" (UID: \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\") "
Oct 30 05:01:01.167697 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.167671    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-openshift-ca\") pod \"a5c7ea18-975b-4f96-80aa-a4f20358d963\" (UID: \"a5c7ea18-975b-4f96-80aa-a4f20358d963\") "
Oct 30 05:01:01.168030 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.168011    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-openshift-ca\") pod \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\" (UID: \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\") "
Oct 30 05:01:01.168227 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.168208    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c7ea18-975b-4f96-80aa-a4f20358d963-kube-api-access-bwmkg\" (UniqueName: \"kubernetes.io/projected/a5c7ea18-975b-4f96-80aa-a4f20358d963-kube-api-access-bwmkg\") pod \"a5c7ea18-975b-4f96-80aa-a4f20358d963\" (UID: \"a5c7ea18-975b-4f96-80aa-a4f20358d963\") "
Oct 30 05:01:01.168450 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.168417    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-kube-api-access-qhp9d\" (UniqueName: \"kubernetes.io/projected/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-kube-api-access-qhp9d\") pod \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\" (UID: \"e595a127-8671-4ba9-a20f-7fbe5d92f3c6\") "
Oct 30 05:01:01.168898 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.168866    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-xks-ca\") pod \"a5c7ea18-975b-4f96-80aa-a4f20358d963\" (UID: \"a5c7ea18-975b-4f96-80aa-a4f20358d963\") "
Oct 30 05:01:01.170467 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.170444    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-openshift-ca" pod "e595a127-8671-4ba9-a20f-7fbe5d92f3c6" (UID: "e595a127-8671-4ba9-a20f-7fbe5d92f3c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:01:01.171593 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.171571    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-openshift-ca" pod "a5c7ea18-975b-4f96-80aa-a4f20358d963" (UID: "a5c7ea18-975b-4f96-80aa-a4f20358d963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:01:01.181260 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.181229    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-xks-ca" pod "e595a127-8671-4ba9-a20f-7fbe5d92f3c6" (UID: "e595a127-8671-4ba9-a20f-7fbe5d92f3c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:01:01.181457 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.181427    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c7ea18-975b-4f96-80aa-a4f20358d963-kube-api-access-bwmkg" pod "a5c7ea18-975b-4f96-80aa-a4f20358d963" (UID: "a5c7ea18-975b-4f96-80aa-a4f20358d963"). InnerVolumeSpecName "kube-api-access-bwmkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:01.181457 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.181443    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-xks-ca" pod "a5c7ea18-975b-4f96-80aa-a4f20358d963" (UID: "a5c7ea18-975b-4f96-80aa-a4f20358d963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:01:01.183272 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.183246    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-kube-api-access-qhp9d" pod "e595a127-8671-4ba9-a20f-7fbe5d92f3c6" (UID: "e595a127-8671-4ba9-a20f-7fbe5d92f3c6"). InnerVolumeSpecName "kube-api-access-qhp9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:01.271356 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.271320    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhp9d\" (UniqueName: \"kubernetes.io/projected/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-kube-api-access-qhp9d\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:01.271356 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.271351    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:01.271356 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.271364    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:01.271608 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.271376    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c7ea18-975b-4f96-80aa-a4f20358d963-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:01.271608 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.271392    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e595a127-8671-4ba9-a20f-7fbe5d92f3c6-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:01.271608 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.271405    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwmkg\" (UniqueName: \"kubernetes.io/projected/a5c7ea18-975b-4f96-80aa-a4f20358d963-kube-api-access-bwmkg\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:01.825694 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.825665    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"
Oct 30 05:01:01.825889 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.825702    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n" event={"ID":"e595a127-8671-4ba9-a20f-7fbe5d92f3c6","Type":"ContainerDied","Data":"1576e27d16d3e7a40b437d8cedb701430d9b7966f6762ec12140b4a8d0681247"}
Oct 30 05:01:01.825889 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.825751    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1576e27d16d3e7a40b437d8cedb701430d9b7966f6762ec12140b4a8d0681247"
Oct 30 05:01:01.827221 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.827191    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"
Oct 30 05:01:01.827336 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.827226    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf" event={"ID":"a5c7ea18-975b-4f96-80aa-a4f20358d963","Type":"ContainerDied","Data":"faff434da252aafe33a0154e906cfea1ed1589e309c1cb94e5d929e0889aadd9"}
Oct 30 05:01:01.827336 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:01.827269    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faff434da252aafe33a0154e906cfea1ed1589e309c1cb94e5d929e0889aadd9"
Oct 30 05:01:02.443727 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:02.443692    5072 scope.go:122] "RemoveContainer" containerID="52833a15d3e59dbf6bd4f3fe6e47514925eb5bb57fee8853b7c0715f45d6c401"
Oct 30 05:01:02.456977 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:02.456951    5072 scope.go:122] "RemoveContainer" containerID="ff3a1ffd1d6e03b9de18ec9deb3581355f370a0fb2d3f72759114f8b5af6b82d"
Oct 30 05:01:04.033799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:04.033765    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"]
Oct 30 05:01:04.046267 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:04.046238    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ebb0e2a-5vz7d"]
Oct 30 05:01:04.411948 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:04.411876    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3b2141-fe42-46f3-bbba-6dfcfd588e1e" path="/var/lib/kubelet/pods/1b3b2141-fe42-46f3-bbba-6dfcfd588e1e/volumes"
Oct 30 05:01:08.790938 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.790887    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"]
Oct 30 05:01:08.792051 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.792025    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a5c7ea18-975b-4f96-80aa-a4f20358d963" containerName="revoke-keys"
Oct 30 05:01:08.792051 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.792051    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c7ea18-975b-4f96-80aa-a4f20358d963" containerName="revoke-keys"
Oct 30 05:01:08.792260 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.792116    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e595a127-8671-4ba9-a20f-7fbe5d92f3c6" containerName="revoke-keys"
Oct 30 05:01:08.792260 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.792124    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e595a127-8671-4ba9-a20f-7fbe5d92f3c6" containerName="revoke-keys"
Oct 30 05:01:08.792385 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.792369    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e595a127-8671-4ba9-a20f-7fbe5d92f3c6" containerName="revoke-keys"
Oct 30 05:01:08.792453 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.792404    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c7ea18-975b-4f96-80aa-a4f20358d963" containerName="revoke-keys"
Oct 30 05:01:08.797449 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.797429    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:08.817068 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.817033    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"]
Oct 30 05:01:08.821984 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.821954    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:01:08.822383 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.822327    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:01:08.892678 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.892644    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a231777e-t86nv\" (UID: \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:08.892867 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.892733    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcl9v\" (UniqueName: \"kubernetes.io/projected/9890a3c3-f015-4c5a-a74d-2d241cb9742b-kube-api-access-lcl9v\") pod \"maas-api-revoke-sub-a231777e-t86nv\" (UID: \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:08.892986 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.892959    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a231777e-t86nv\" (UID: \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:08.994687 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.994655    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a231777e-t86nv\" (UID: \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:08.994873 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.994741    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcl9v\" (UniqueName: \"kubernetes.io/projected/9890a3c3-f015-4c5a-a74d-2d241cb9742b-kube-api-access-lcl9v\") pod \"maas-api-revoke-sub-a231777e-t86nv\" (UID: \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:08.994928 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.994896    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a231777e-t86nv\" (UID: \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:08.995561 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.995533    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a231777e-t86nv\" (UID: \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:08.997360 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:08.997330    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a231777e-t86nv\" (UID: \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:09.018873 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:09.018848    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcl9v\" (UniqueName: \"kubernetes.io/projected/9890a3c3-f015-4c5a-a74d-2d241cb9742b-kube-api-access-lcl9v\") pod \"maas-api-revoke-sub-a231777e-t86nv\" (UID: \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:09.114495 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:09.114418    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:09.444753 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:01:09.444705    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9890a3c3_f015_4c5a_a74d_2d241cb9742b.slice/crio-46130da2c7aa3e711c723c699192a1604ccfc5ebfe7848107ee8b39ac5b6ec78 WatchSource:0}: Error finding container 46130da2c7aa3e711c723c699192a1604ccfc5ebfe7848107ee8b39ac5b6ec78: Status 404 returned error can't find the container with id 46130da2c7aa3e711c723c699192a1604ccfc5ebfe7848107ee8b39ac5b6ec78
Oct 30 05:01:09.445345 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:09.445319    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"]
Oct 30 05:01:09.901811 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:09.901779    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv" event={"ID":"9890a3c3-f015-4c5a-a74d-2d241cb9742b","Type":"ContainerStarted","Data":"b667f8100d90d356c9f9cf8bc76c4d26fb4d69f5645e1aab912ff1d1c53b2842"}
Oct 30 05:01:09.902262 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:09.901830    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv" event={"ID":"9890a3c3-f015-4c5a-a74d-2d241cb9742b","Type":"ContainerStarted","Data":"46130da2c7aa3e711c723c699192a1604ccfc5ebfe7848107ee8b39ac5b6ec78"}
Oct 30 05:01:09.949210 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:09.949114    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv" podStartSLOduration=1.949096588 podStartE2EDuration="1.949096588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:09.947367405 +0000 UTC m=+3849.688790506" watchObservedRunningTime="2026-10-30 05:01:09.949096588 +0000 UTC m=+3849.690519682"
Oct 30 05:01:10.697031 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:10.696999    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"]
Oct 30 05:01:10.917344 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:10.916855    5072 generic.go:356] "Generic (PLEG): container finished" podID="9890a3c3-f015-4c5a-a74d-2d241cb9742b" containerID="b667f8100d90d356c9f9cf8bc76c4d26fb4d69f5645e1aab912ff1d1c53b2842" exitCode=0
Oct 30 05:01:10.917731 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:10.916950    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv" event={"ID":"9890a3c3-f015-4c5a-a74d-2d241cb9742b","Type":"ContainerDied","Data":"b667f8100d90d356c9f9cf8bc76c4d26fb4d69f5645e1aab912ff1d1c53b2842"}
Oct 30 05:01:11.232220 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.232181    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"]
Oct 30 05:01:11.238869 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.238835    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:11.273849 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.272135    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"]
Oct 30 05:01:11.346583 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.346538    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz76m\" (UniqueName: \"kubernetes.io/projected/c413afde-5c6c-43b2-af90-49ad91009908-kube-api-access-vz76m\") pod \"maas-api-revoke-sub-69bdb975-vtbzp\" (UID: \"c413afde-5c6c-43b2-af90-49ad91009908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:11.346767 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.346623    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69bdb975-vtbzp\" (UID: \"c413afde-5c6c-43b2-af90-49ad91009908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:11.346850 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.346825    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69bdb975-vtbzp\" (UID: \"c413afde-5c6c-43b2-af90-49ad91009908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:11.448895 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.448860    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69bdb975-vtbzp\" (UID: \"c413afde-5c6c-43b2-af90-49ad91009908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:11.449066 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.449036    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69bdb975-vtbzp\" (UID: \"c413afde-5c6c-43b2-af90-49ad91009908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:11.449181 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.449168    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz76m\" (UniqueName: \"kubernetes.io/projected/c413afde-5c6c-43b2-af90-49ad91009908-kube-api-access-vz76m\") pod \"maas-api-revoke-sub-69bdb975-vtbzp\" (UID: \"c413afde-5c6c-43b2-af90-49ad91009908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:11.449633 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.449613    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69bdb975-vtbzp\" (UID: \"c413afde-5c6c-43b2-af90-49ad91009908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:11.451653 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.451626    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69bdb975-vtbzp\" (UID: \"c413afde-5c6c-43b2-af90-49ad91009908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:11.473335 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.473307    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz76m\" (UniqueName: \"kubernetes.io/projected/c413afde-5c6c-43b2-af90-49ad91009908-kube-api-access-vz76m\") pod \"maas-api-revoke-sub-69bdb975-vtbzp\" (UID: \"c413afde-5c6c-43b2-af90-49ad91009908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:11.573552 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.573477    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:11.752682 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:01:11.752644    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:01:11.888290 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:01:11.888257    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc413afde_5c6c_43b2_af90_49ad91009908.slice/crio-2b92a1b0970e57ea1a9d9a3c4ea6d59539474181b5c5198b5c70c122ddcda348 WatchSource:0}: Error finding container 2b92a1b0970e57ea1a9d9a3c4ea6d59539474181b5c5198b5c70c122ddcda348: Status 404 returned error can't find the container with id 2b92a1b0970e57ea1a9d9a3c4ea6d59539474181b5c5198b5c70c122ddcda348
Oct 30 05:01:11.889421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.889396    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"]
Oct 30 05:01:11.927841 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:11.927816    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp" event={"ID":"c413afde-5c6c-43b2-af90-49ad91009908","Type":"ContainerStarted","Data":"2b92a1b0970e57ea1a9d9a3c4ea6d59539474181b5c5198b5c70c122ddcda348"}
Oct 30 05:01:12.113896 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.113858    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:12.278894 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.278856    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9890a3c3-f015-4c5a-a74d-2d241cb9742b-kube-api-access-lcl9v\" (UniqueName: \"kubernetes.io/projected/9890a3c3-f015-4c5a-a74d-2d241cb9742b-kube-api-access-lcl9v\") pod \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\" (UID: \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\") "
Oct 30 05:01:12.279118 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.279036    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-openshift-ca\") pod \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\" (UID: \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\") "
Oct 30 05:01:12.279201 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.279177    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-xks-ca\") pod \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\" (UID: \"9890a3c3-f015-4c5a-a74d-2d241cb9742b\") "
Oct 30 05:01:12.279425 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.279394    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-openshift-ca" pod "9890a3c3-f015-4c5a-a74d-2d241cb9742b" (UID: "9890a3c3-f015-4c5a-a74d-2d241cb9742b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:01:12.279799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.279778    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:12.281209 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.281183    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9890a3c3-f015-4c5a-a74d-2d241cb9742b-kube-api-access-lcl9v" pod "9890a3c3-f015-4c5a-a74d-2d241cb9742b" (UID: "9890a3c3-f015-4c5a-a74d-2d241cb9742b"). InnerVolumeSpecName "kube-api-access-lcl9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:12.281335 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.281318    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-xks-ca" pod "9890a3c3-f015-4c5a-a74d-2d241cb9742b" (UID: "9890a3c3-f015-4c5a-a74d-2d241cb9742b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:01:12.381180 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.381151    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9890a3c3-f015-4c5a-a74d-2d241cb9742b-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:12.381180 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.381176    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcl9v\" (UniqueName: \"kubernetes.io/projected/9890a3c3-f015-4c5a-a74d-2d241cb9742b-kube-api-access-lcl9v\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:12.938333 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.938304    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"
Oct 30 05:01:12.938333 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.938310    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv" event={"ID":"9890a3c3-f015-4c5a-a74d-2d241cb9742b","Type":"ContainerDied","Data":"46130da2c7aa3e711c723c699192a1604ccfc5ebfe7848107ee8b39ac5b6ec78"}
Oct 30 05:01:12.938814 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.938369    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46130da2c7aa3e711c723c699192a1604ccfc5ebfe7848107ee8b39ac5b6ec78"
Oct 30 05:01:12.940164 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.940141    5072 generic.go:356] "Generic (PLEG): container finished" podID="c413afde-5c6c-43b2-af90-49ad91009908" containerID="7b45e055b8774f46ad46c12f0ecba1e6b867d7ebdaa2a544d1e9320cb0b78803" exitCode=0
Oct 30 05:01:12.940283 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:12.940219    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp" event={"ID":"c413afde-5c6c-43b2-af90-49ad91009908","Type":"ContainerDied","Data":"7b45e055b8774f46ad46c12f0ecba1e6b867d7ebdaa2a544d1e9320cb0b78803"}
Oct 30 05:01:14.035353 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.035314    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"]
Oct 30 05:01:14.045392 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.045358    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ea49f6-sw4m2"]
Oct 30 05:01:14.121989 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.121961    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:14.207096 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.206983    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-xks-ca\") pod \"c413afde-5c6c-43b2-af90-49ad91009908\" (UID: \"c413afde-5c6c-43b2-af90-49ad91009908\") "
Oct 30 05:01:14.207280 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.207151    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-openshift-ca\") pod \"c413afde-5c6c-43b2-af90-49ad91009908\" (UID: \"c413afde-5c6c-43b2-af90-49ad91009908\") "
Oct 30 05:01:14.207280 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.207257    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c413afde-5c6c-43b2-af90-49ad91009908-kube-api-access-vz76m\" (UniqueName: \"kubernetes.io/projected/c413afde-5c6c-43b2-af90-49ad91009908-kube-api-access-vz76m\") pod \"c413afde-5c6c-43b2-af90-49ad91009908\" (UID: \"c413afde-5c6c-43b2-af90-49ad91009908\") "
Oct 30 05:01:14.207608 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.207577    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-openshift-ca" pod "c413afde-5c6c-43b2-af90-49ad91009908" (UID: "c413afde-5c6c-43b2-af90-49ad91009908"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:01:14.207964 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.207939    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:14.209627 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.209592    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c413afde-5c6c-43b2-af90-49ad91009908-kube-api-access-vz76m" pod "c413afde-5c6c-43b2-af90-49ad91009908" (UID: "c413afde-5c6c-43b2-af90-49ad91009908"). InnerVolumeSpecName "kube-api-access-vz76m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:14.209891 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.209868    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-xks-ca" pod "c413afde-5c6c-43b2-af90-49ad91009908" (UID: "c413afde-5c6c-43b2-af90-49ad91009908"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:01:14.308923 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.308889    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz76m\" (UniqueName: \"kubernetes.io/projected/c413afde-5c6c-43b2-af90-49ad91009908-kube-api-access-vz76m\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:14.308923 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.308921    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c413afde-5c6c-43b2-af90-49ad91009908-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:14.412193 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.412162    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e57e6b3-15af-4562-87de-48093607a0f4" path="/var/lib/kubelet/pods/0e57e6b3-15af-4562-87de-48093607a0f4/volumes"
Oct 30 05:01:14.959780 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.959750    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"
Oct 30 05:01:14.959987 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.959830    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp" event={"ID":"c413afde-5c6c-43b2-af90-49ad91009908","Type":"ContainerDied","Data":"2b92a1b0970e57ea1a9d9a3c4ea6d59539474181b5c5198b5c70c122ddcda348"}
Oct 30 05:01:14.959987 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:14.959886    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b92a1b0970e57ea1a9d9a3c4ea6d59539474181b5c5198b5c70c122ddcda348"
Oct 30 05:01:25.052886 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:25.052849    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"]
Oct 30 05:01:25.066989 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:25.066960    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af82feed-wprqp"]
Oct 30 05:01:25.726821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:25.726782    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4" podUID="ac55f48a-97c9-4178-8e54-8b12063986e6" containerName="main" containerID="cri-o://2e5362e0ddf8c4a163c5a56b9e3d20c94cc1c9dae2a83e7e1e1ef53f8e0db51d" gracePeriod=45
Oct 30 05:01:26.054111 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.054043    5072 generic.go:356] "Generic (PLEG): container finished" podID="ac55f48a-97c9-4178-8e54-8b12063986e6" containerID="2e5362e0ddf8c4a163c5a56b9e3d20c94cc1c9dae2a83e7e1e1ef53f8e0db51d" exitCode=0
Oct 30 05:01:26.054540 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.054109    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4" event={"ID":"ac55f48a-97c9-4178-8e54-8b12063986e6","Type":"ContainerDied","Data":"2e5362e0ddf8c4a163c5a56b9e3d20c94cc1c9dae2a83e7e1e1ef53f8e0db51d"}
Oct 30 05:01:26.054540 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.054147    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4" event={"ID":"ac55f48a-97c9-4178-8e54-8b12063986e6","Type":"ContainerDied","Data":"d3a3e1b0b20f76d7254be43710e05c44a3e6acab1d0f2d2429e24f89fb3af8a7"}
Oct 30 05:01:26.054540 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.054159    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3a3e1b0b20f76d7254be43710e05c44a3e6acab1d0f2d2429e24f89fb3af8a7"
Oct 30 05:01:26.056220 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.056201    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 05:01:26.181016 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.180987    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-dshm\") pod \"ac55f48a-97c9-4178-8e54-8b12063986e6\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") "
Oct 30 05:01:26.181196 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.181053    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-tmp-dir\") pod \"ac55f48a-97c9-4178-8e54-8b12063986e6\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") "
Oct 30 05:01:26.181196 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.181132    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac55f48a-97c9-4178-8e54-8b12063986e6-kube-api-access-zj2cg\" (UniqueName: \"kubernetes.io/projected/ac55f48a-97c9-4178-8e54-8b12063986e6-kube-api-access-zj2cg\") pod \"ac55f48a-97c9-4178-8e54-8b12063986e6\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") "
Oct 30 05:01:26.181196 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.181152    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-home\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-home\") pod \"ac55f48a-97c9-4178-8e54-8b12063986e6\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") "
Oct 30 05:01:26.181196 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.181179    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-model-cache\") pod \"ac55f48a-97c9-4178-8e54-8b12063986e6\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") "
Oct 30 05:01:26.181447 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.181286    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac55f48a-97c9-4178-8e54-8b12063986e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac55f48a-97c9-4178-8e54-8b12063986e6-tls-certs\") pod \"ac55f48a-97c9-4178-8e54-8b12063986e6\" (UID: \"ac55f48a-97c9-4178-8e54-8b12063986e6\") "
Oct 30 05:01:26.181447 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.181325    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-tmp-dir" pod "ac55f48a-97c9-4178-8e54-8b12063986e6" (UID: "ac55f48a-97c9-4178-8e54-8b12063986e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:26.181551 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.181449    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-home" pod "ac55f48a-97c9-4178-8e54-8b12063986e6" (UID: "ac55f48a-97c9-4178-8e54-8b12063986e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:26.181551 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.181498    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-model-cache" pod "ac55f48a-97c9-4178-8e54-8b12063986e6" (UID: "ac55f48a-97c9-4178-8e54-8b12063986e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:26.182007 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.181928    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:26.182007 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.181948    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:26.182007 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.181958    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:26.183470 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.183438    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-dshm" pod "ac55f48a-97c9-4178-8e54-8b12063986e6" (UID: "ac55f48a-97c9-4178-8e54-8b12063986e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:26.183579 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.183508    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac55f48a-97c9-4178-8e54-8b12063986e6-tls-certs" pod "ac55f48a-97c9-4178-8e54-8b12063986e6" (UID: "ac55f48a-97c9-4178-8e54-8b12063986e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:01:26.183579 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.183553    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac55f48a-97c9-4178-8e54-8b12063986e6-kube-api-access-zj2cg" pod "ac55f48a-97c9-4178-8e54-8b12063986e6" (UID: "ac55f48a-97c9-4178-8e54-8b12063986e6"). InnerVolumeSpecName "kube-api-access-zj2cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:26.283839 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.283774    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac55f48a-97c9-4178-8e54-8b12063986e6-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:26.283839 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.283803    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac55f48a-97c9-4178-8e54-8b12063986e6-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:26.283839 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.283812    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj2cg\" (UniqueName: \"kubernetes.io/projected/ac55f48a-97c9-4178-8e54-8b12063986e6-kube-api-access-zj2cg\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:26.410894 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.410857    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2775b776-88da-49f2-81ad-2bd8a68b3023" path="/var/lib/kubelet/pods/2775b776-88da-49f2-81ad-2bd8a68b3023/volumes"
Oct 30 05:01:26.935242 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:26.935196    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4" podUID="ac55f48a-97c9-4178-8e54-8b12063986e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:27.062132 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:27.062096    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"
Oct 30 05:01:27.120993 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:27.120957    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"]
Oct 30 05:01:27.134390 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:27.134359    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5f1508/test-model-5f1508-kserve-9cd4cfb5-kf9f4"]
Oct 30 05:01:28.412504 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:28.412473    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac55f48a-97c9-4178-8e54-8b12063986e6" path="/var/lib/kubelet/pods/ac55f48a-97c9-4178-8e54-8b12063986e6/volumes"
Oct 30 05:01:31.398409 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.398372    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"]
Oct 30 05:01:31.399702 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.399681    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ac55f48a-97c9-4178-8e54-8b12063986e6" containerName="main"
Oct 30 05:01:31.399769 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.399704    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac55f48a-97c9-4178-8e54-8b12063986e6" containerName="main"
Oct 30 05:01:31.399769 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.399732    5072 cpu_manager.go:414] "containerMap: removing container" podUID="9890a3c3-f015-4c5a-a74d-2d241cb9742b" containerName="revoke-keys"
Oct 30 05:01:31.399769 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.399740    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9890a3c3-f015-4c5a-a74d-2d241cb9742b" containerName="revoke-keys"
Oct 30 05:01:31.399891 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.399783    5072 cpu_manager.go:414] "containerMap: removing container" podUID="c413afde-5c6c-43b2-af90-49ad91009908" containerName="revoke-keys"
Oct 30 05:01:31.399891 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.399791    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c413afde-5c6c-43b2-af90-49ad91009908" containerName="revoke-keys"
Oct 30 05:01:31.400025 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.400011    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c413afde-5c6c-43b2-af90-49ad91009908" containerName="revoke-keys"
Oct 30 05:01:31.400061 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.400038    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac55f48a-97c9-4178-8e54-8b12063986e6" containerName="main"
Oct 30 05:01:31.400061 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.400047    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9890a3c3-f015-4c5a-a74d-2d241cb9742b" containerName="revoke-keys"
Oct 30 05:01:31.406141 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.406113    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.450254 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.450198    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-26b72f\"/\"kube-root-ca.crt\""
Oct 30 05:01:31.450254 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.450249    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-26b72f\"/\"openshift-service-ca.crt\""
Oct 30 05:01:31.450497 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.450380    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-26b72f\"/\"test-model-26b72f-kserve-self-signed-certs\""
Oct 30 05:01:31.450561 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.450496    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-26b72f\"/\"default-dockercfg-m8rgx\""
Oct 30 05:01:31.463645 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.463611    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"]
Oct 30 05:01:31.483602 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.483571    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-dshm\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.483762 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.483739    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tmp-dir\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.483927 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.483910    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-model-cache\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.484190 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.484168    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-home\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.484306 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.484220    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tls-certs\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.484390 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.484340    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwj92\" (UniqueName: \"kubernetes.io/projected/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-kube-api-access-gwj92\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.585768 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.585724    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-dshm\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.585975 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.585828    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tmp-dir\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.585975 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.585941    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-model-cache\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.586175 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.586154    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-home\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.586259 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.586199    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tls-certs\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.586259 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.586241    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwj92\" (UniqueName: \"kubernetes.io/projected/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-kube-api-access-gwj92\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.586370 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.586254    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tmp-dir\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.586370 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.586295    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-model-cache\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.586697 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.586672    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-home\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.588463 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.588428    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-dshm\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.589489 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.589467    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tls-certs\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.613751 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.613728    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwj92\" (UniqueName: \"kubernetes.io/projected/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-kube-api-access-gwj92\") pod \"test-model-26b72f-kserve-7f9878c5d5-nj8gl\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") " pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:31.722027 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:31.721997    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:32.151405 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:01:32.151364    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bf3c58a_bfdf_4225_9cc5_90f6e10d5639.slice/crio-07e70cce0c5960173272ca8809d99e43b3a5a45770fad37fa83e5bd18ce508cc WatchSource:0}: Error finding container 07e70cce0c5960173272ca8809d99e43b3a5a45770fad37fa83e5bd18ce508cc: Status 404 returned error can't find the container with id 07e70cce0c5960173272ca8809d99e43b3a5a45770fad37fa83e5bd18ce508cc
Oct 30 05:01:32.152134 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:32.152102    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"]
Oct 30 05:01:33.118546 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:33.118513    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl" event={"ID":"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639","Type":"ContainerStarted","Data":"7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415"}
Oct 30 05:01:33.119062 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:33.118557    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl" event={"ID":"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639","Type":"ContainerStarted","Data":"07e70cce0c5960173272ca8809d99e43b3a5a45770fad37fa83e5bd18ce508cc"}
Oct 30 05:01:33.198749 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:33.198695    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl" podStartSLOduration=2.198680444 podStartE2EDuration="2.198680444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:33.194808732 +0000 UTC m=+3872.936231830" watchObservedRunningTime="2026-10-30 05:01:33.198680444 +0000 UTC m=+3872.940103540"
Oct 30 05:01:33.723056 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:33.723023    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:33.976598 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:33.976515    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"]
Oct 30 05:01:38.052278 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:38.052244    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"]
Oct 30 05:01:38.063570 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:38.063542    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e135cd6-v8sr7"]
Oct 30 05:01:38.410733 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:38.410659    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35684429-00ad-418e-b11b-a8f5b1c0caef" path="/var/lib/kubelet/pods/35684429-00ad-418e-b11b-a8f5b1c0caef/volumes"
Oct 30 05:01:41.722335 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:41.722301    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:41.734937 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:41.734909    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:41.753561 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:01:41.753526    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:01:42.211351 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:42.211323    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:01:43.524701 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:43.524669    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"]
Oct 30 05:01:43.559112 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:43.559041    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"]
Oct 30 05:01:44.313345 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:44.313310    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"]
Oct 30 05:01:44.353601 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:44.353572    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"]
Oct 30 05:01:48.031815 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:48.031779    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"]
Oct 30 05:01:48.043729 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:48.043703    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb71983a-5hqgj"]
Oct 30 05:01:48.411382 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:48.411297    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39aac01-debe-4cb2-81c4-90e41bf627b9" path="/var/lib/kubelet/pods/b39aac01-debe-4cb2-81c4-90e41bf627b9/volumes"
Oct 30 05:01:48.525629 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:48.525591    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb" podUID="737a04c9-e996-4325-b019-7fab7930e0ac" containerName="payload-processing" containerID="cri-o://0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7" gracePeriod=25
Oct 30 05:01:48.560316 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:48.560281    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx" podUID="e141e106-199b-4c45-aee5-2b7492ed4fb4" containerName="payload-pre-processing" containerID="cri-o://5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747" gracePeriod=25
Oct 30 05:01:49.003190 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.003135    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh" podUID="092d6318-b45b-40d4-b3ec-7b2ad2769e04" containerName="main" containerID="cri-o://51aa9cd35ea1d353aa9c13384ad7fb15dafebab3b456d082fa015d512bbd1da0" gracePeriod=45
Oct 30 05:01:49.284960 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.284926    5072 generic.go:356] "Generic (PLEG): container finished" podID="092d6318-b45b-40d4-b3ec-7b2ad2769e04" containerID="51aa9cd35ea1d353aa9c13384ad7fb15dafebab3b456d082fa015d512bbd1da0" exitCode=0
Oct 30 05:01:49.285441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.285003    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh" event={"ID":"092d6318-b45b-40d4-b3ec-7b2ad2769e04","Type":"ContainerDied","Data":"51aa9cd35ea1d353aa9c13384ad7fb15dafebab3b456d082fa015d512bbd1da0"}
Oct 30 05:01:49.285441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.285043    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh" event={"ID":"092d6318-b45b-40d4-b3ec-7b2ad2769e04","Type":"ContainerDied","Data":"bc0595e722c726d621b60b38978718e06402ceba02d2c0c75f92e54a68266eab"}
Oct 30 05:01:49.285441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.285058    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc0595e722c726d621b60b38978718e06402ceba02d2c0c75f92e54a68266eab"
Oct 30 05:01:49.301332 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.301307    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 05:01:49.314039 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.314006    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5" podUID="fc63106e-36a8-498e-9807-3676ef0dcb1c" containerName="payload-processing" containerID="cri-o://263c6e7f2faac6a3be51eb45b78b7594c175175f220e647bf726e3761a951ec6" gracePeriod=25
Oct 30 05:01:49.353915 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.353868    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw" podUID="0fcdf75f-0261-4af4-b574-5048789081f6" containerName="payload-pre-processing" containerID="cri-o://51f2cb80618a41001899b8ce1be4ec7a69df60fbb85b81588a4f9395f2738dbd" gracePeriod=25
Oct 30 05:01:49.384927 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.384894    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092d6318-b45b-40d4-b3ec-7b2ad2769e04-kube-api-access-672x8\" (UniqueName: \"kubernetes.io/projected/092d6318-b45b-40d4-b3ec-7b2ad2769e04-kube-api-access-672x8\") pod \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") "
Oct 30 05:01:49.385149 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.384955    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-model-cache\") pod \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") "
Oct 30 05:01:49.385238 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.385169    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-dshm\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-dshm\") pod \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") "
Oct 30 05:01:49.385238 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.385198    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tls-certs\" (UniqueName: \"kubernetes.io/secret/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tls-certs\") pod \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") "
Oct 30 05:01:49.385352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.385254    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-home\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-home\") pod \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") "
Oct 30 05:01:49.385352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.385303    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tmp-dir\") pod \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\" (UID: \"092d6318-b45b-40d4-b3ec-7b2ad2769e04\") "
Oct 30 05:01:49.385352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.385308    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-model-cache" pod "092d6318-b45b-40d4-b3ec-7b2ad2769e04" (UID: "092d6318-b45b-40d4-b3ec-7b2ad2769e04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:49.385675 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.385646    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-home" pod "092d6318-b45b-40d4-b3ec-7b2ad2769e04" (UID: "092d6318-b45b-40d4-b3ec-7b2ad2769e04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:49.385830 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.385785    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tmp-dir" pod "092d6318-b45b-40d4-b3ec-7b2ad2769e04" (UID: "092d6318-b45b-40d4-b3ec-7b2ad2769e04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:49.386229 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.386207    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:49.386330 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.386233    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:49.386330 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.386246    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:49.389028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.389001    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-dshm" pod "092d6318-b45b-40d4-b3ec-7b2ad2769e04" (UID: "092d6318-b45b-40d4-b3ec-7b2ad2769e04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:49.389592 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.389557    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092d6318-b45b-40d4-b3ec-7b2ad2769e04-kube-api-access-672x8" pod "092d6318-b45b-40d4-b3ec-7b2ad2769e04" (UID: "092d6318-b45b-40d4-b3ec-7b2ad2769e04"). InnerVolumeSpecName "kube-api-access-672x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:49.389806 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.389782    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tls-certs" pod "092d6318-b45b-40d4-b3ec-7b2ad2769e04" (UID: "092d6318-b45b-40d4-b3ec-7b2ad2769e04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:01:49.488033 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.488003    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092d6318-b45b-40d4-b3ec-7b2ad2769e04-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:49.488033 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.488030    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092d6318-b45b-40d4-b3ec-7b2ad2769e04-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:49.488033 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:49.488040    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-672x8\" (UniqueName: \"kubernetes.io/projected/092d6318-b45b-40d4-b3ec-7b2ad2769e04-kube-api-access-672x8\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:01:50.294271 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:50.294239    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"
Oct 30 05:01:50.350532 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:50.350502    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"]
Oct 30 05:01:50.359730 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:50.359690    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-82bec5/test-model-82bec5-kserve-56f6b87d77-vhfbh"]
Oct 30 05:01:50.411907 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:50.411872    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="092d6318-b45b-40d4-b3ec-7b2ad2769e04" path="/var/lib/kubelet/pods/092d6318-b45b-40d4-b3ec-7b2ad2769e04/volumes"
Oct 30 05:01:52.078238 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:52.078202    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw" podUID="0fcdf75f-0261-4af4-b574-5048789081f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:01:52.078621 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:52.078202    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5" podUID="fc63106e-36a8-498e-9807-3676ef0dcb1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:01:54.932402 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:54.932365    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx" podUID="e141e106-199b-4c45-aee5-2b7492ed4fb4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:01:55.940412 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:55.940371    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb" podUID="737a04c9-e996-4325-b019-7fab7930e0ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:01:56.732773 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:56.732733    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"]
Oct 30 05:01:56.733018 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:01:56.732985    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf" podUID="9d0da867-beb3-412d-9af4-128b6c39dbdf" containerName="istio-proxy" containerID="cri-o://fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514" gracePeriod=30
Oct 30 05:02:01.004708 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:01.004674    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:02:01.023147 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:01.023124    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:02:02.050324 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.050296    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 05:02:02.078463 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.078430    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw" podUID="0fcdf75f-0261-4af4-b574-5048789081f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:02:02.078616 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.078439    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5" podUID="fc63106e-36a8-498e-9807-3676ef0dcb1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:02:02.170352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.170315    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-socket\") pod \"9d0da867-beb3-412d-9af4-128b6c39dbdf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") "
Oct 30 05:02:02.170575 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.170487    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-envoy\") pod \"9d0da867-beb3-412d-9af4-128b6c39dbdf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") "
Oct 30 05:02:02.170575 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.170564    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-credential-socket\") pod \"9d0da867-beb3-412d-9af4-128b6c39dbdf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") "
Oct 30 05:02:02.170720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.170603    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-socket" pod "9d0da867-beb3-412d-9af4-128b6c39dbdf" (UID: "9d0da867-beb3-412d-9af4-128b6c39dbdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:02.170720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.170657    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-kube-api-access-z6glz\" (UniqueName: \"kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-kube-api-access-z6glz\") pod \"9d0da867-beb3-412d-9af4-128b6c39dbdf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") "
Oct 30 05:02:02.170949 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.170913    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-credential-socket" pod "9d0da867-beb3-412d-9af4-128b6c39dbdf" (UID: "9d0da867-beb3-412d-9af4-128b6c39dbdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:02.171188 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.171168    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-data\") pod \"9d0da867-beb3-412d-9af4-128b6c39dbdf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") "
Oct 30 05:02:02.171298 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.171228    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-podinfo\") pod \"9d0da867-beb3-412d-9af4-128b6c39dbdf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") "
Oct 30 05:02:02.171298 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.171289    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0da867-beb3-412d-9af4-128b6c39dbdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0da867-beb3-412d-9af4-128b6c39dbdf-istiod-ca-cert\") pod \"9d0da867-beb3-412d-9af4-128b6c39dbdf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") "
Oct 30 05:02:02.171422 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.171361    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-token\" (UniqueName: \"kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-token\") pod \"9d0da867-beb3-412d-9af4-128b6c39dbdf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") "
Oct 30 05:02:02.171553 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.171532    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-certs\") pod \"9d0da867-beb3-412d-9af4-128b6c39dbdf\" (UID: \"9d0da867-beb3-412d-9af4-128b6c39dbdf\") "
Oct 30 05:02:02.171735 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.171710    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-data" pod "9d0da867-beb3-412d-9af4-128b6c39dbdf" (UID: "9d0da867-beb3-412d-9af4-128b6c39dbdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:02.172342 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.172103    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0da867-beb3-412d-9af4-128b6c39dbdf-istiod-ca-cert" pod "9d0da867-beb3-412d-9af4-128b6c39dbdf" (UID: "9d0da867-beb3-412d-9af4-128b6c39dbdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:02.172342 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.172273    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:02.172342 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.172292    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:02.172342 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.172306    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:02.172342 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.172319    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0da867-beb3-412d-9af4-128b6c39dbdf-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:02.173327 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.173043    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-certs" pod "9d0da867-beb3-412d-9af4-128b6c39dbdf" (UID: "9d0da867-beb3-412d-9af4-128b6c39dbdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:02.173541 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.173524    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-envoy" pod "9d0da867-beb3-412d-9af4-128b6c39dbdf" (UID: "9d0da867-beb3-412d-9af4-128b6c39dbdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:02.174336 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.174307    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-podinfo" pod "9d0da867-beb3-412d-9af4-128b6c39dbdf" (UID: "9d0da867-beb3-412d-9af4-128b6c39dbdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:02:02.174650 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.174632    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-token" pod "9d0da867-beb3-412d-9af4-128b6c39dbdf" (UID: "9d0da867-beb3-412d-9af4-128b6c39dbdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:02.175553 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.175531    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-kube-api-access-z6glz" pod "9d0da867-beb3-412d-9af4-128b6c39dbdf" (UID: "9d0da867-beb3-412d-9af4-128b6c39dbdf"). InnerVolumeSpecName "kube-api-access-z6glz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:02.273441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.273398    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6glz\" (UniqueName: \"kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-kube-api-access-z6glz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:02.273441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.273435    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:02.273441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.273448    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:02.273688 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.273459    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:02.273688 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.273472    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0da867-beb3-412d-9af4-128b6c39dbdf-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:02.401697 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.401621    5072 generic.go:356] "Generic (PLEG): container finished" podID="9d0da867-beb3-412d-9af4-128b6c39dbdf" containerID="fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514" exitCode=0
Oct 30 05:02:02.401847 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.401726    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"
Oct 30 05:02:02.411946 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.411921    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf" event={"ID":"9d0da867-beb3-412d-9af4-128b6c39dbdf","Type":"ContainerDied","Data":"fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514"}
Oct 30 05:02:02.412041 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.411966    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf" event={"ID":"9d0da867-beb3-412d-9af4-128b6c39dbdf","Type":"ContainerDied","Data":"d9b3da27a2b2f8ca8d6ba4b6107eeb4511cc0c005e476d846346e25badeb4555"}
Oct 30 05:02:02.412041 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.411989    5072 scope.go:122] "RemoveContainer" containerID="fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514"
Oct 30 05:02:02.424593 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.424560    5072 scope.go:122] "RemoveContainer" containerID="fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514"
Oct 30 05:02:02.424862 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:02:02.424840    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514\": container with ID starting with fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514 not found: ID does not exist" containerID="fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514"
Oct 30 05:02:02.424934 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.424873    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514"} err="failed to get container status \"fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514\": rpc error: code = NotFound desc = could not find container \"fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514\": container with ID starting with fb3f24bcf395800ea3642dc12dc74d332c0a4dd5b982283a5abb523d53f9f514 not found: ID does not exist"
Oct 30 05:02:02.458831 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.458807    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"]
Oct 30 05:02:02.474469 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.474447    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-7npbf"]
Oct 30 05:02:02.647145 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.647116    5072 scope.go:122] "RemoveContainer" containerID="5dc8845bdd13a3e58633f62b081342d22ace08b7941d3eb76cb6deebf0fe0b8c"
Oct 30 05:02:02.662846 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:02.662819    5072 scope.go:122] "RemoveContainer" containerID="1117e6b6a49a81a73d8e259857400e016c5bb596bba62e6cf3da94b63da32c97"
Oct 30 05:02:04.411045 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:04.411008    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0da867-beb3-412d-9af4-128b6c39dbdf" path="/var/lib/kubelet/pods/9d0da867-beb3-412d-9af4-128b6c39dbdf/volumes"
Oct 30 05:02:04.932717 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:04.932676    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx" podUID="e141e106-199b-4c45-aee5-2b7492ed4fb4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:02:05.940861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:05.940826    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb" podUID="737a04c9-e996-4325-b019-7fab7930e0ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:02:08.908948 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:08.908919    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"
Oct 30 05:02:08.912826 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:08.912804    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"
Oct 30 05:02:08.975465 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:08.975421    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737a04c9-e996-4325-b019-7fab7930e0ac-kube-api-access-qbbkr\" (UniqueName: \"kubernetes.io/projected/737a04c9-e996-4325-b019-7fab7930e0ac-kube-api-access-qbbkr\") pod \"737a04c9-e996-4325-b019-7fab7930e0ac\" (UID: \"737a04c9-e996-4325-b019-7fab7930e0ac\") "
Oct 30 05:02:08.976177 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:08.976151    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e141e106-199b-4c45-aee5-2b7492ed4fb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e141e106-199b-4c45-aee5-2b7492ed4fb4-plugins-config-volume\") pod \"e141e106-199b-4c45-aee5-2b7492ed4fb4\" (UID: \"e141e106-199b-4c45-aee5-2b7492ed4fb4\") "
Oct 30 05:02:08.976985 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:08.976866    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e141e106-199b-4c45-aee5-2b7492ed4fb4-plugins-config-volume" pod "e141e106-199b-4c45-aee5-2b7492ed4fb4" (UID: "e141e106-199b-4c45-aee5-2b7492ed4fb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:08.977166 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:08.977147    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/737a04c9-e996-4325-b019-7fab7930e0ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737a04c9-e996-4325-b019-7fab7930e0ac-plugins-config-volume\") pod \"737a04c9-e996-4325-b019-7fab7930e0ac\" (UID: \"737a04c9-e996-4325-b019-7fab7930e0ac\") "
Oct 30 05:02:08.977834 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:08.977712    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737a04c9-e996-4325-b019-7fab7930e0ac-plugins-config-volume" pod "737a04c9-e996-4325-b019-7fab7930e0ac" (UID: "737a04c9-e996-4325-b019-7fab7930e0ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:08.978403 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:08.978066    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e141e106-199b-4c45-aee5-2b7492ed4fb4-kube-api-access-hfqgz\" (UniqueName: \"kubernetes.io/projected/e141e106-199b-4c45-aee5-2b7492ed4fb4-kube-api-access-hfqgz\") pod \"e141e106-199b-4c45-aee5-2b7492ed4fb4\" (UID: \"e141e106-199b-4c45-aee5-2b7492ed4fb4\") "
Oct 30 05:02:08.979388 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:08.979366    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e141e106-199b-4c45-aee5-2b7492ed4fb4-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:08.980057 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:08.979519    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737a04c9-e996-4325-b019-7fab7930e0ac-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:08.983985 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:08.982839    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e141e106-199b-4c45-aee5-2b7492ed4fb4-kube-api-access-hfqgz" pod "e141e106-199b-4c45-aee5-2b7492ed4fb4" (UID: "e141e106-199b-4c45-aee5-2b7492ed4fb4"). InnerVolumeSpecName "kube-api-access-hfqgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:08.992759 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:08.992730    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737a04c9-e996-4325-b019-7fab7930e0ac-kube-api-access-qbbkr" pod "737a04c9-e996-4325-b019-7fab7930e0ac" (UID: "737a04c9-e996-4325-b019-7fab7930e0ac"). InnerVolumeSpecName "kube-api-access-qbbkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:09.082487 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.082448    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfqgz\" (UniqueName: \"kubernetes.io/projected/e141e106-199b-4c45-aee5-2b7492ed4fb4-kube-api-access-hfqgz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:09.082487 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.082481    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbbkr\" (UniqueName: \"kubernetes.io/projected/737a04c9-e996-4325-b019-7fab7930e0ac-kube-api-access-qbbkr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:09.477941 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.477902    5072 generic.go:356] "Generic (PLEG): container finished" podID="e141e106-199b-4c45-aee5-2b7492ed4fb4" containerID="5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747" exitCode=0
Oct 30 05:02:09.478104 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.478008    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"
Oct 30 05:02:09.478104 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.478018    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx" event={"ID":"e141e106-199b-4c45-aee5-2b7492ed4fb4","Type":"ContainerDied","Data":"5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747"}
Oct 30 05:02:09.478258 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.478100    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx" event={"ID":"e141e106-199b-4c45-aee5-2b7492ed4fb4","Type":"ContainerDied","Data":"bd857dcfd05019d84d74fe98b92a95652670411ba8d500faefa4501ba3c7b193"}
Oct 30 05:02:09.478258 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.478135    5072 scope.go:122] "RemoveContainer" containerID="5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747"
Oct 30 05:02:09.480231 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.480206    5072 generic.go:356] "Generic (PLEG): container finished" podID="737a04c9-e996-4325-b019-7fab7930e0ac" containerID="0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7" exitCode=0
Oct 30 05:02:09.480350 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.480308    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"
Oct 30 05:02:09.480350 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.480324    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb" event={"ID":"737a04c9-e996-4325-b019-7fab7930e0ac","Type":"ContainerDied","Data":"0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7"}
Oct 30 05:02:09.480513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.480379    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb" event={"ID":"737a04c9-e996-4325-b019-7fab7930e0ac","Type":"ContainerDied","Data":"9d8434e68fd983fe7f0fc79b12983b8d0ef953aeca6ac054d075f3ed7cc4130e"}
Oct 30 05:02:09.483345 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.483321    5072 generic.go:356] "Generic (PLEG): container finished" podID="fc63106e-36a8-498e-9807-3676ef0dcb1c" containerID="263c6e7f2faac6a3be51eb45b78b7594c175175f220e647bf726e3761a951ec6" exitCode=0
Oct 30 05:02:09.483507 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.483396    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5" event={"ID":"fc63106e-36a8-498e-9807-3676ef0dcb1c","Type":"ContainerDied","Data":"263c6e7f2faac6a3be51eb45b78b7594c175175f220e647bf726e3761a951ec6"}
Oct 30 05:02:09.486109 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.486047    5072 generic.go:356] "Generic (PLEG): container finished" podID="0fcdf75f-0261-4af4-b574-5048789081f6" containerID="51f2cb80618a41001899b8ce1be4ec7a69df60fbb85b81588a4f9395f2738dbd" exitCode=0
Oct 30 05:02:09.486228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.486152    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw" event={"ID":"0fcdf75f-0261-4af4-b574-5048789081f6","Type":"ContainerDied","Data":"51f2cb80618a41001899b8ce1be4ec7a69df60fbb85b81588a4f9395f2738dbd"}
Oct 30 05:02:09.495566 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.495548    5072 scope.go:122] "RemoveContainer" containerID="5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747"
Oct 30 05:02:09.495890 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:02:09.495866    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747\": container with ID starting with 5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747 not found: ID does not exist" containerID="5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747"
Oct 30 05:02:09.496015 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.495898    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747"} err="failed to get container status \"5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747\": rpc error: code = NotFound desc = could not find container \"5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747\": container with ID starting with 5968e83e1316f32164e33ea4b611696d7d82ab2b1f1f4d8f5abf09297501f747 not found: ID does not exist"
Oct 30 05:02:09.496015 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.495921    5072 scope.go:122] "RemoveContainer" containerID="0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7"
Oct 30 05:02:09.508142 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.508121    5072 scope.go:122] "RemoveContainer" containerID="0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7"
Oct 30 05:02:09.508547 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:02:09.508525    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7\": container with ID starting with 0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7 not found: ID does not exist" containerID="0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7"
Oct 30 05:02:09.508632 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.508578    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7"} err="failed to get container status \"0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7\": rpc error: code = NotFound desc = could not find container \"0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7\": container with ID starting with 0ddd1171d253ae41d404cccf764ee884305fffe44d62f8a5c084beb79e5a1af7 not found: ID does not exist"
Oct 30 05:02:09.567121 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.564430    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"]
Oct 30 05:02:09.567121 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.566632    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f1508-768b74fdc4-xpvnb"]
Oct 30 05:02:09.603221 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.597605    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"]
Oct 30 05:02:09.613410 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.613379    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f1508-6bfd88d9f5-nvjqx"]
Oct 30 05:02:09.695037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.694977    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"
Oct 30 05:02:09.743294 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.743266    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"
Oct 30 05:02:09.792673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.790956    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fcdf75f-0261-4af4-b574-5048789081f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fcdf75f-0261-4af4-b574-5048789081f6-plugins-config-volume\") pod \"0fcdf75f-0261-4af4-b574-5048789081f6\" (UID: \"0fcdf75f-0261-4af4-b574-5048789081f6\") "
Oct 30 05:02:09.792673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.791042    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc63106e-36a8-498e-9807-3676ef0dcb1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc63106e-36a8-498e-9807-3676ef0dcb1c-plugins-config-volume\") pod \"fc63106e-36a8-498e-9807-3676ef0dcb1c\" (UID: \"fc63106e-36a8-498e-9807-3676ef0dcb1c\") "
Oct 30 05:02:09.792673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.791112    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc63106e-36a8-498e-9807-3676ef0dcb1c-kube-api-access-wbx9b\" (UniqueName: \"kubernetes.io/projected/fc63106e-36a8-498e-9807-3676ef0dcb1c-kube-api-access-wbx9b\") pod \"fc63106e-36a8-498e-9807-3676ef0dcb1c\" (UID: \"fc63106e-36a8-498e-9807-3676ef0dcb1c\") "
Oct 30 05:02:09.792673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.791226    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fcdf75f-0261-4af4-b574-5048789081f6-kube-api-access-f8hjt\" (UniqueName: \"kubernetes.io/projected/0fcdf75f-0261-4af4-b574-5048789081f6-kube-api-access-f8hjt\") pod \"0fcdf75f-0261-4af4-b574-5048789081f6\" (UID: \"0fcdf75f-0261-4af4-b574-5048789081f6\") "
Oct 30 05:02:09.792673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.791356    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fcdf75f-0261-4af4-b574-5048789081f6-plugins-config-volume" pod "0fcdf75f-0261-4af4-b574-5048789081f6" (UID: "0fcdf75f-0261-4af4-b574-5048789081f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:09.792673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.791811    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc63106e-36a8-498e-9807-3676ef0dcb1c-plugins-config-volume" pod "fc63106e-36a8-498e-9807-3676ef0dcb1c" (UID: "fc63106e-36a8-498e-9807-3676ef0dcb1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:09.792673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.791920    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fcdf75f-0261-4af4-b574-5048789081f6-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:09.792673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.791938    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc63106e-36a8-498e-9807-3676ef0dcb1c-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:09.797511 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.797487    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fcdf75f-0261-4af4-b574-5048789081f6-kube-api-access-f8hjt" pod "0fcdf75f-0261-4af4-b574-5048789081f6" (UID: "0fcdf75f-0261-4af4-b574-5048789081f6"). InnerVolumeSpecName "kube-api-access-f8hjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:09.797642 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.797582    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc63106e-36a8-498e-9807-3676ef0dcb1c-kube-api-access-wbx9b" pod "fc63106e-36a8-498e-9807-3676ef0dcb1c" (UID: "fc63106e-36a8-498e-9807-3676ef0dcb1c"). InnerVolumeSpecName "kube-api-access-wbx9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:09.892859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.892779    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbx9b\" (UniqueName: \"kubernetes.io/projected/fc63106e-36a8-498e-9807-3676ef0dcb1c-kube-api-access-wbx9b\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:09.892859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:09.892810    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8hjt\" (UniqueName: \"kubernetes.io/projected/0fcdf75f-0261-4af4-b574-5048789081f6-kube-api-access-f8hjt\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:10.410985 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:10.410951    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="737a04c9-e996-4325-b019-7fab7930e0ac" path="/var/lib/kubelet/pods/737a04c9-e996-4325-b019-7fab7930e0ac/volumes"
Oct 30 05:02:10.411498 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:10.411476    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e141e106-199b-4c45-aee5-2b7492ed4fb4" path="/var/lib/kubelet/pods/e141e106-199b-4c45-aee5-2b7492ed4fb4/volumes"
Oct 30 05:02:10.496226 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:10.496197    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"
Oct 30 05:02:10.496359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:10.496225    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw" event={"ID":"0fcdf75f-0261-4af4-b574-5048789081f6","Type":"ContainerDied","Data":"d05e2d80d892971e9730504f73846a806ccca5e1a59b802d448e13129ecf4434"}
Oct 30 05:02:10.496359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:10.496279    5072 scope.go:122] "RemoveContainer" containerID="51f2cb80618a41001899b8ce1be4ec7a69df60fbb85b81588a4f9395f2738dbd"
Oct 30 05:02:10.500577 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:10.500547    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5" event={"ID":"fc63106e-36a8-498e-9807-3676ef0dcb1c","Type":"ContainerDied","Data":"cce7e83a94602ed8f61e2e342b8ff7e22dd84829ab127ac27dcd7ef4ea91f2a0"}
Oct 30 05:02:10.500699 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:10.500616    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"
Oct 30 05:02:10.511898 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:10.511875    5072 scope.go:122] "RemoveContainer" containerID="263c6e7f2faac6a3be51eb45b78b7594c175175f220e647bf726e3761a951ec6"
Oct 30 05:02:10.552684 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:10.552663    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"]
Oct 30 05:02:10.565483 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:10.565458    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-134dad-7bcdc5b659-djmlw"]
Oct 30 05:02:10.605303 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:10.605269    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"]
Oct 30 05:02:10.614899 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:10.614862    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-134dad-6c4f8498cd-w5bc5"]
Oct 30 05:02:11.753507 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:02:11.753469    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:02:12.411224 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:12.411191    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fcdf75f-0261-4af4-b574-5048789081f6" path="/var/lib/kubelet/pods/0fcdf75f-0261-4af4-b574-5048789081f6/volumes"
Oct 30 05:02:12.411625 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:12.411610    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc63106e-36a8-498e-9807-3676ef0dcb1c" path="/var/lib/kubelet/pods/fc63106e-36a8-498e-9807-3676ef0dcb1c/volumes"
Oct 30 05:02:12.508667 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:12.508636    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"]
Oct 30 05:02:12.508890 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:12.508865    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" podUID="b735ef67-f82e-44b6-9306-fdad4e8ef244" containerName="istio-proxy" containerID="cri-o://785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948" gracePeriod=30
Oct 30 05:02:14.184435 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:14.184394    5072 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:02:14.184841 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:14.184468    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" podUID="b735ef67-f82e-44b6-9306-fdad4e8ef244" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:02:17.821955 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.821925    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 05:02:17.931117 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.931062    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-socket\") pod \"b735ef67-f82e-44b6-9306-fdad4e8ef244\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") "
Oct 30 05:02:17.931333 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.931162    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-credential-socket\") pod \"b735ef67-f82e-44b6-9306-fdad4e8ef244\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") "
Oct 30 05:02:17.931427 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.931401    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-envoy\") pod \"b735ef67-f82e-44b6-9306-fdad4e8ef244\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") "
Oct 30 05:02:17.931502 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.931423    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-socket" pod "b735ef67-f82e-44b6-9306-fdad4e8ef244" (UID: "b735ef67-f82e-44b6-9306-fdad4e8ef244"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:17.931565 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.931504    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-credential-socket" pod "b735ef67-f82e-44b6-9306-fdad4e8ef244" (UID: "b735ef67-f82e-44b6-9306-fdad4e8ef244"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:17.931626 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.931585    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-token\" (UniqueName: \"kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-token\") pod \"b735ef67-f82e-44b6-9306-fdad4e8ef244\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") "
Oct 30 05:02:17.931690 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.931651    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-certs\") pod \"b735ef67-f82e-44b6-9306-fdad4e8ef244\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") "
Oct 30 05:02:17.931750 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.931694    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-data\") pod \"b735ef67-f82e-44b6-9306-fdad4e8ef244\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") "
Oct 30 05:02:17.931810 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.931750    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-podinfo\") pod \"b735ef67-f82e-44b6-9306-fdad4e8ef244\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") "
Oct 30 05:02:17.931869 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.931828    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-kube-api-access-86w4l\" (UniqueName: \"kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-kube-api-access-86w4l\") pod \"b735ef67-f82e-44b6-9306-fdad4e8ef244\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") "
Oct 30 05:02:17.931929 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.931872    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b735ef67-f82e-44b6-9306-fdad4e8ef244-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b735ef67-f82e-44b6-9306-fdad4e8ef244-istiod-ca-cert\") pod \"b735ef67-f82e-44b6-9306-fdad4e8ef244\" (UID: \"b735ef67-f82e-44b6-9306-fdad4e8ef244\") "
Oct 30 05:02:17.932578 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.932547    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-data" pod "b735ef67-f82e-44b6-9306-fdad4e8ef244" (UID: "b735ef67-f82e-44b6-9306-fdad4e8ef244"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:17.932718 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.932648    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b735ef67-f82e-44b6-9306-fdad4e8ef244-istiod-ca-cert" pod "b735ef67-f82e-44b6-9306-fdad4e8ef244" (UID: "b735ef67-f82e-44b6-9306-fdad4e8ef244"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:17.932718 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.932689    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-certs" pod "b735ef67-f82e-44b6-9306-fdad4e8ef244" (UID: "b735ef67-f82e-44b6-9306-fdad4e8ef244"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:17.933152 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.933129    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:17.933234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.933158    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:17.933234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.933172    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b735ef67-f82e-44b6-9306-fdad4e8ef244-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:17.933234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.933186    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:17.933234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.933198    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:17.934474 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.934447    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-envoy" pod "b735ef67-f82e-44b6-9306-fdad4e8ef244" (UID: "b735ef67-f82e-44b6-9306-fdad4e8ef244"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:17.934474 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.934462    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-token" pod "b735ef67-f82e-44b6-9306-fdad4e8ef244" (UID: "b735ef67-f82e-44b6-9306-fdad4e8ef244"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:17.934945 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.934720    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-kube-api-access-86w4l" pod "b735ef67-f82e-44b6-9306-fdad4e8ef244" (UID: "b735ef67-f82e-44b6-9306-fdad4e8ef244"). InnerVolumeSpecName "kube-api-access-86w4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:17.935058 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:17.934992    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-podinfo" pod "b735ef67-f82e-44b6-9306-fdad4e8ef244" (UID: "b735ef67-f82e-44b6-9306-fdad4e8ef244"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:02:18.034720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.034684    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:18.034720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.034714    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:18.034720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.034724    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86w4l\" (UniqueName: \"kubernetes.io/projected/b735ef67-f82e-44b6-9306-fdad4e8ef244-kube-api-access-86w4l\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:18.035123 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.034734    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b735ef67-f82e-44b6-9306-fdad4e8ef244-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:18.622913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.622879    5072 generic.go:356] "Generic (PLEG): container finished" podID="b735ef67-f82e-44b6-9306-fdad4e8ef244" containerID="785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948" exitCode=0
Oct 30 05:02:18.623151 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.622943    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"
Oct 30 05:02:18.623151 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.622963    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" event={"ID":"b735ef67-f82e-44b6-9306-fdad4e8ef244","Type":"ContainerDied","Data":"785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948"}
Oct 30 05:02:18.623151 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.623022    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d" event={"ID":"b735ef67-f82e-44b6-9306-fdad4e8ef244","Type":"ContainerDied","Data":"7fdf3dfcf53dc13f3c4a52abf16851fd45caa1e2ce5dae7e2ed57142b45087b5"}
Oct 30 05:02:18.623151 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.623048    5072 scope.go:122] "RemoveContainer" containerID="785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948"
Oct 30 05:02:18.638815 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.638601    5072 scope.go:122] "RemoveContainer" containerID="785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948"
Oct 30 05:02:18.639014 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:02:18.638993    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948\": container with ID starting with 785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948 not found: ID does not exist" containerID="785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948"
Oct 30 05:02:18.639065 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.639023    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948"} err="failed to get container status \"785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948\": rpc error: code = NotFound desc = could not find container \"785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948\": container with ID starting with 785e4d14bdfaf79d0c59c1246ffe2b68eb279d531d0b3eda1f35e34218c5c948 not found: ID does not exist"
Oct 30 05:02:18.692680 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.692633    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"]
Oct 30 05:02:18.706522 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:18.706489    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-f5n6d"]
Oct 30 05:02:20.410241 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:20.410209    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b735ef67-f82e-44b6-9306-fdad4e8ef244" path="/var/lib/kubelet/pods/b735ef67-f82e-44b6-9306-fdad4e8ef244/volumes"
Oct 30 05:02:23.845587 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:23.845556    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"]
Oct 30 05:02:28.846481 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:28.846445    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw" podUID="66af17d7-50b8-4277-984a-a5fb20858c56" containerName="payload-pre-processing" containerID="cri-o://6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013" gracePeriod=25
Oct 30 05:02:29.954649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:29.954614    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw" podUID="66af17d7-50b8-4277-984a-a5fb20858c56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:02:32.388607 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.388572    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"]
Oct 30 05:02:32.389755 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389734    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e141e106-199b-4c45-aee5-2b7492ed4fb4" containerName="payload-pre-processing"
Oct 30 05:02:32.389821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389757    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e141e106-199b-4c45-aee5-2b7492ed4fb4" containerName="payload-pre-processing"
Oct 30 05:02:32.389821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389780    5072 cpu_manager.go:414] "containerMap: removing container" podUID="092d6318-b45b-40d4-b3ec-7b2ad2769e04" containerName="main"
Oct 30 05:02:32.389821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389786    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="092d6318-b45b-40d4-b3ec-7b2ad2769e04" containerName="main"
Oct 30 05:02:32.389821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389801    5072 cpu_manager.go:414] "containerMap: removing container" podUID="737a04c9-e996-4325-b019-7fab7930e0ac" containerName="payload-processing"
Oct 30 05:02:32.389821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389805    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737a04c9-e996-4325-b019-7fab7930e0ac" containerName="payload-processing"
Oct 30 05:02:32.389821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389814    5072 cpu_manager.go:414] "containerMap: removing container" podUID="9d0da867-beb3-412d-9af4-128b6c39dbdf" containerName="istio-proxy"
Oct 30 05:02:32.389821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389818    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0da867-beb3-412d-9af4-128b6c39dbdf" containerName="istio-proxy"
Oct 30 05:02:32.390037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389843    5072 cpu_manager.go:414] "containerMap: removing container" podUID="fc63106e-36a8-498e-9807-3676ef0dcb1c" containerName="payload-processing"
Oct 30 05:02:32.390037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389847    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc63106e-36a8-498e-9807-3676ef0dcb1c" containerName="payload-processing"
Oct 30 05:02:32.390037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389857    5072 cpu_manager.go:414] "containerMap: removing container" podUID="b735ef67-f82e-44b6-9306-fdad4e8ef244" containerName="istio-proxy"
Oct 30 05:02:32.390037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389861    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b735ef67-f82e-44b6-9306-fdad4e8ef244" containerName="istio-proxy"
Oct 30 05:02:32.390037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389873    5072 cpu_manager.go:414] "containerMap: removing container" podUID="0fcdf75f-0261-4af4-b574-5048789081f6" containerName="payload-pre-processing"
Oct 30 05:02:32.390037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.389877    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fcdf75f-0261-4af4-b574-5048789081f6" containerName="payload-pre-processing"
Oct 30 05:02:32.390037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.390001    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e141e106-199b-4c45-aee5-2b7492ed4fb4" containerName="payload-pre-processing"
Oct 30 05:02:32.390037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.390014    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="092d6318-b45b-40d4-b3ec-7b2ad2769e04" containerName="main"
Oct 30 05:02:32.390037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.390022    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="737a04c9-e996-4325-b019-7fab7930e0ac" containerName="payload-processing"
Oct 30 05:02:32.390037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.390027    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0da867-beb3-412d-9af4-128b6c39dbdf" containerName="istio-proxy"
Oct 30 05:02:32.390037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.390036    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc63106e-36a8-498e-9807-3676ef0dcb1c" containerName="payload-processing"
Oct 30 05:02:32.390037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.390041    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b735ef67-f82e-44b6-9306-fdad4e8ef244" containerName="istio-proxy"
Oct 30 05:02:32.390473 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.390048    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fcdf75f-0261-4af4-b574-5048789081f6" containerName="payload-pre-processing"
Oct 30 05:02:32.395769 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.395746    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.416050 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.416022    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"]
Oct 30 05:02:32.423374 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.423351    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-26b72f-openshift-default-dockercfg-555w7\""
Oct 30 05:02:32.441261 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.441224    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-socket\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.441261 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.441256    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3bd8c9-443a-45a3-9965-02c57f42164b-istiod-ca-cert\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.441462 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.441356    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-envoy\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.441462 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.441388    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-token\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.441462 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.441444    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-certs\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.441656 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.441471    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt79z\" (UniqueName: \"kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-kube-api-access-rt79z\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.441656 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.441557    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-data\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.441656 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.441648    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-credential-socket\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.441833 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.441766    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-podinfo\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.542767 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.542732    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-podinfo\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.542936 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.542786    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-socket\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.542936 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.542911    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3bd8c9-443a-45a3-9965-02c57f42164b-istiod-ca-cert\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.543068 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.542987    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-envoy\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.543068 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.543025    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-token\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.543239 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.543117    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-certs\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.543239 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.543144    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt79z\" (UniqueName: \"kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-kube-api-access-rt79z\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.543239 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.543178    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-socket\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.543410 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.543291    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-data\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.543499 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.543472    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-credential-socket\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.543728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.543641    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-data\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.543728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.543693    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3bd8c9-443a-45a3-9965-02c57f42164b-istiod-ca-cert\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.543728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.543696    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-certs\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.543970 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.543893    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-credential-socket\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.545813 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.545787    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-podinfo\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.546139 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.546116    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-envoy\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.566985 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.566961    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt79z\" (UniqueName: \"kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-kube-api-access-rt79z\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.567119 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.567063    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-token\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:32.715086 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:32.715055    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:33.045373 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:02:33.045307    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3bd8c9_443a_45a3_9965_02c57f42164b.slice/crio-86bd6468cf73ca91ac2d6e9638014173f1fa562ac26b75317a24e8bc7967f7af WatchSource:0}: Error finding container 86bd6468cf73ca91ac2d6e9638014173f1fa562ac26b75317a24e8bc7967f7af: Status 404 returned error can't find the container with id 86bd6468cf73ca91ac2d6e9638014173f1fa562ac26b75317a24e8bc7967f7af
Oct 30 05:02:33.046419 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:33.046391    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"]
Oct 30 05:02:33.048739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:33.048706    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:02:33.048827 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:33.048771    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:02:33.048827 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:33.048802    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:02:33.788784 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:33.788745    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm" event={"ID":"ba3bd8c9-443a-45a3-9965-02c57f42164b","Type":"ContainerStarted","Data":"c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4"}
Oct 30 05:02:33.789280 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:33.788803    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm" event={"ID":"ba3bd8c9-443a-45a3-9965-02c57f42164b","Type":"ContainerStarted","Data":"86bd6468cf73ca91ac2d6e9638014173f1fa562ac26b75317a24e8bc7967f7af"}
Oct 30 05:02:33.893158 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:33.893106    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm" podStartSLOduration=1.893090666 podStartE2EDuration="1.893090666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:33.890512917 +0000 UTC m=+3933.631936025" watchObservedRunningTime="2026-10-30 05:02:33.893090666 +0000 UTC m=+3933.634513754"
Oct 30 05:02:34.716138 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:34.716101    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:34.717732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:34.717702    5072 patch_prober.go:29] interesting pod/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 05:02:34.717873 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:34.717766    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm" podUID="ba3bd8c9-443a-45a3-9965-02c57f42164b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 05:02:35.716029 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:35.715985    5072 patch_prober.go:29] interesting pod/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 05:02:35.716512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:35.716058    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm" podUID="ba3bd8c9-443a-45a3-9965-02c57f42164b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 05:02:36.730001 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:36.727525    5072 patch_prober.go:29] interesting pod/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 05:02:36.730001 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:36.727602    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm" podUID="ba3bd8c9-443a-45a3-9965-02c57f42164b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 05:02:37.715484 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:37.715443    5072 patch_prober.go:29] interesting pod/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 05:02:37.715677 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:37.715499    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm" podUID="ba3bd8c9-443a-45a3-9965-02c57f42164b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 05:02:38.726101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:38.725572    5072 patch_prober.go:29] interesting pod/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 05:02:38.726101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:38.725652    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm" podUID="ba3bd8c9-443a-45a3-9965-02c57f42164b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 05:02:39.725234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:39.725202    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:39.725777 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:39.725753    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:39.726845 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:39.726824    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:02:39.958232 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:39.955593    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw" podUID="66af17d7-50b8-4277-984a-a5fb20858c56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:02:40.006952 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:40.006874    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"]
Oct 30 05:02:40.007409 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:40.007379    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz" podUID="d9d290bd-4133-45ce-9816-a7b94e8d89a4" containerName="istio-proxy" containerID="cri-o://cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8" gracePeriod=30
Oct 30 05:02:41.753163 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:02:41.753125    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:02:45.326504 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.326477    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 05:02:45.431799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.431760    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-envoy\") pod \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") "
Oct 30 05:02:45.432005 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.431821    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-certs\") pod \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") "
Oct 30 05:02:45.432110 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.431903    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-kube-api-access-lvqvz\" (UniqueName: \"kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-kube-api-access-lvqvz\") pod \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") "
Oct 30 05:02:45.432190 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.432161    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-podinfo\") pod \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") "
Oct 30 05:02:45.432266 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.432249    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-token\" (UniqueName: \"kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-token\") pod \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") "
Oct 30 05:02:45.432334 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.432311    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-data\") pod \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") "
Oct 30 05:02:45.432618 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.432593    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-socket\") pod \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") "
Oct 30 05:02:45.432732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.432672    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-credential-socket\") pod \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") "
Oct 30 05:02:45.432732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.432713    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istiod-ca-cert\") pod \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\" (UID: \"d9d290bd-4133-45ce-9816-a7b94e8d89a4\") "
Oct 30 05:02:45.435720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.435689    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istiod-ca-cert" pod "d9d290bd-4133-45ce-9816-a7b94e8d89a4" (UID: "d9d290bd-4133-45ce-9816-a7b94e8d89a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:45.444761 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.444506    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-data" pod "d9d290bd-4133-45ce-9816-a7b94e8d89a4" (UID: "d9d290bd-4133-45ce-9816-a7b94e8d89a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:45.445214 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.444923    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-socket" pod "d9d290bd-4133-45ce-9816-a7b94e8d89a4" (UID: "d9d290bd-4133-45ce-9816-a7b94e8d89a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:45.446402 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.445342    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-credential-socket" pod "d9d290bd-4133-45ce-9816-a7b94e8d89a4" (UID: "d9d290bd-4133-45ce-9816-a7b94e8d89a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:45.446402 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.446387    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-certs" pod "d9d290bd-4133-45ce-9816-a7b94e8d89a4" (UID: "d9d290bd-4133-45ce-9816-a7b94e8d89a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:45.448765 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.448734    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-kube-api-access-lvqvz" pod "d9d290bd-4133-45ce-9816-a7b94e8d89a4" (UID: "d9d290bd-4133-45ce-9816-a7b94e8d89a4"). InnerVolumeSpecName "kube-api-access-lvqvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:45.458432 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.458323    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-podinfo" pod "d9d290bd-4133-45ce-9816-a7b94e8d89a4" (UID: "d9d290bd-4133-45ce-9816-a7b94e8d89a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:02:45.461135 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.461106    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-token" pod "d9d290bd-4133-45ce-9816-a7b94e8d89a4" (UID: "d9d290bd-4133-45ce-9816-a7b94e8d89a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:45.466428 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.466395    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-envoy" pod "d9d290bd-4133-45ce-9816-a7b94e8d89a4" (UID: "d9d290bd-4133-45ce-9816-a7b94e8d89a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:45.536674 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.536642    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.536674 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.536669    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.536852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.536682    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.536852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.536693    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.536852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.536705    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.536852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.536717    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.536852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.536731    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.536852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.536742    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d290bd-4133-45ce-9816-a7b94e8d89a4-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.536852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.536755    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvqvz\" (UniqueName: \"kubernetes.io/projected/d9d290bd-4133-45ce-9816-a7b94e8d89a4-kube-api-access-lvqvz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.901206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.901174    5072 generic.go:356] "Generic (PLEG): container finished" podID="d9d290bd-4133-45ce-9816-a7b94e8d89a4" containerID="cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8" exitCode=0
Oct 30 05:02:45.901384 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.901242    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"
Oct 30 05:02:45.901384 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.901257    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz" event={"ID":"d9d290bd-4133-45ce-9816-a7b94e8d89a4","Type":"ContainerDied","Data":"cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8"}
Oct 30 05:02:45.901384 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.901349    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz" event={"ID":"d9d290bd-4133-45ce-9816-a7b94e8d89a4","Type":"ContainerDied","Data":"7ff76620fb44fa946a18dc1881750d0857b667ee694f93ded61b6afd99e04007"}
Oct 30 05:02:45.901384 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.901381    5072 scope.go:122] "RemoveContainer" containerID="cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8"
Oct 30 05:02:45.913485 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.913464    5072 scope.go:122] "RemoveContainer" containerID="cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8"
Oct 30 05:02:45.913744 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:02:45.913725    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8\": container with ID starting with cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8 not found: ID does not exist" containerID="cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8"
Oct 30 05:02:45.913817 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.913752    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8"} err="failed to get container status \"cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8\": rpc error: code = NotFound desc = could not find container \"cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8\": container with ID starting with cdba41e985f0523555e6f7a8320fac6d2213e576685fc6c9b4865e54e650c4f8 not found: ID does not exist"
Oct 30 05:02:45.970754 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.970724    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"]
Oct 30 05:02:45.983393 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:45.983369    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-54qmz"]
Oct 30 05:02:46.410638 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:46.410602    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d290bd-4133-45ce-9816-a7b94e8d89a4" path="/var/lib/kubelet/pods/d9d290bd-4133-45ce-9816-a7b94e8d89a4/volumes"
Oct 30 05:02:46.870218 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:46.870187    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"]
Oct 30 05:02:46.910061 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:46.910031    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"]
Oct 30 05:02:49.148332 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.148309    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"
Oct 30 05:02:49.301197 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.301155    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66af17d7-50b8-4277-984a-a5fb20858c56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66af17d7-50b8-4277-984a-a5fb20858c56-plugins-config-volume\") pod \"66af17d7-50b8-4277-984a-a5fb20858c56\" (UID: \"66af17d7-50b8-4277-984a-a5fb20858c56\") "
Oct 30 05:02:49.301425 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.301409    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66af17d7-50b8-4277-984a-a5fb20858c56-kube-api-access-qb95w\" (UniqueName: \"kubernetes.io/projected/66af17d7-50b8-4277-984a-a5fb20858c56-kube-api-access-qb95w\") pod \"66af17d7-50b8-4277-984a-a5fb20858c56\" (UID: \"66af17d7-50b8-4277-984a-a5fb20858c56\") "
Oct 30 05:02:49.301599 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.301571    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66af17d7-50b8-4277-984a-a5fb20858c56-plugins-config-volume" pod "66af17d7-50b8-4277-984a-a5fb20858c56" (UID: "66af17d7-50b8-4277-984a-a5fb20858c56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:49.301911 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.301890    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66af17d7-50b8-4277-984a-a5fb20858c56-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:49.304223 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.304189    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66af17d7-50b8-4277-984a-a5fb20858c56-kube-api-access-qb95w" pod "66af17d7-50b8-4277-984a-a5fb20858c56" (UID: "66af17d7-50b8-4277-984a-a5fb20858c56"). InnerVolumeSpecName "kube-api-access-qb95w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:49.403035 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.402934    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb95w\" (UniqueName: \"kubernetes.io/projected/66af17d7-50b8-4277-984a-a5fb20858c56-kube-api-access-qb95w\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:02:49.939251 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.939216    5072 generic.go:356] "Generic (PLEG): container finished" podID="66af17d7-50b8-4277-984a-a5fb20858c56" containerID="6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013" exitCode=0
Oct 30 05:02:49.939421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.939282    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"
Oct 30 05:02:49.939421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.939302    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw" event={"ID":"66af17d7-50b8-4277-984a-a5fb20858c56","Type":"ContainerDied","Data":"6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013"}
Oct 30 05:02:49.939421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.939360    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw" event={"ID":"66af17d7-50b8-4277-984a-a5fb20858c56","Type":"ContainerDied","Data":"1815e0265f103ec4547f69afa259101a11567cf2bddd49014567f10397dca90b"}
Oct 30 05:02:49.939421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.939387    5072 scope.go:122] "RemoveContainer" containerID="6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013"
Oct 30 05:02:49.958764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.958734    5072 scope.go:122] "RemoveContainer" containerID="6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013"
Oct 30 05:02:49.959050 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:02:49.959020    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013\": container with ID starting with 6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013 not found: ID does not exist" containerID="6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013"
Oct 30 05:02:49.959238 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.959060    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013"} err="failed to get container status \"6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013\": rpc error: code = NotFound desc = could not find container \"6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013\": container with ID starting with 6e68ed1316843e8f64f8f25019f8995f76a790ac8d911a655adb73b778037013 not found: ID does not exist"
Oct 30 05:02:49.999603 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:49.999571    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"]
Oct 30 05:02:50.011847 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:50.011815    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82bec5-6896b4d959-ssqtw"]
Oct 30 05:02:50.412752 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:50.412664    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66af17d7-50b8-4277-984a-a5fb20858c56" path="/var/lib/kubelet/pods/66af17d7-50b8-4277-984a-a5fb20858c56/volumes"
Oct 30 05:02:51.870993 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:51.870951    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc" podUID="476eab0b-9b8a-449a-a65d-5dadf7f4483b" containerName="payload-processing" containerID="cri-o://79419a5814aa013c46bac61324f1fc71cb99c415f641ebc74a66cc1f8abd4ba5" gracePeriod=25
Oct 30 05:02:51.910855 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:51.910815    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt" podUID="2bf7fdb4-1e6b-4575-a470-e659bd8306ed" containerName="payload-pre-processing" containerID="cri-o://c0990b8667af174f479e81ee134f54c93276fcaf1af96f7c70b3fb7609653210" gracePeriod=25
Oct 30 05:02:59.729443 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:02:59.729401    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt" podUID="2bf7fdb4-1e6b-4575-a470-e659bd8306ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:03:00.737344 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:00.737298    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc" podUID="476eab0b-9b8a-449a-a65d-5dadf7f4483b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:03:02.705527 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:02.705494    5072 scope.go:122] "RemoveContainer" containerID="c5b0af732d8fcb8ec11e56cec8f9f93a0cce88d1f4d4fda917e91491899587af"
Oct 30 05:03:02.716937 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:02.716917    5072 scope.go:122] "RemoveContainer" containerID="51aa9cd35ea1d353aa9c13384ad7fb15dafebab3b456d082fa015d512bbd1da0"
Oct 30 05:03:02.726638 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:02.726614    5072 scope.go:122] "RemoveContainer" containerID="9924e1db6442003f228ed6e2a6b2c28fdab8d0bd7b7e6c3fa2c8bc4cfbbe773c"
Oct 30 05:03:02.736173 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:02.736149    5072 scope.go:122] "RemoveContainer" containerID="b753111e05e32220613dca70e6d89e6b39555567d35b7504fe9062c261959bad"
Oct 30 05:03:02.748325 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:02.748301    5072 scope.go:122] "RemoveContainer" containerID="2e5362e0ddf8c4a163c5a56b9e3d20c94cc1c9dae2a83e7e1e1ef53f8e0db51d"
Oct 30 05:03:02.758107 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:02.758055    5072 scope.go:122] "RemoveContainer" containerID="1763d8a7d3f57ca8ae045ed7c06a1be1817610c344e24e49f86f3742a43ab7d7"
Oct 30 05:03:09.728615 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:09.728578    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt" podUID="2bf7fdb4-1e6b-4575-a470-e659bd8306ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:03:10.737061 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:10.737017    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc" podUID="476eab0b-9b8a-449a-a65d-5dadf7f4483b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:03:11.753474 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:03:11.753437    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:03:12.125923 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.125884    5072 generic.go:356] "Generic (PLEG): container finished" podID="476eab0b-9b8a-449a-a65d-5dadf7f4483b" containerID="79419a5814aa013c46bac61324f1fc71cb99c415f641ebc74a66cc1f8abd4ba5" exitCode=0
Oct 30 05:03:12.126137 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.126050    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc" event={"ID":"476eab0b-9b8a-449a-a65d-5dadf7f4483b","Type":"ContainerDied","Data":"79419a5814aa013c46bac61324f1fc71cb99c415f641ebc74a66cc1f8abd4ba5"}
Oct 30 05:03:12.128389 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.128361    5072 generic.go:356] "Generic (PLEG): container finished" podID="2bf7fdb4-1e6b-4575-a470-e659bd8306ed" containerID="c0990b8667af174f479e81ee134f54c93276fcaf1af96f7c70b3fb7609653210" exitCode=0
Oct 30 05:03:12.128671 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.128556    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt" event={"ID":"2bf7fdb4-1e6b-4575-a470-e659bd8306ed","Type":"ContainerDied","Data":"c0990b8667af174f479e81ee134f54c93276fcaf1af96f7c70b3fb7609653210"}
Oct 30 05:03:12.208678 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.208649    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"
Oct 30 05:03:12.233735 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.233705    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476eab0b-9b8a-449a-a65d-5dadf7f4483b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476eab0b-9b8a-449a-a65d-5dadf7f4483b-plugins-config-volume\") pod \"476eab0b-9b8a-449a-a65d-5dadf7f4483b\" (UID: \"476eab0b-9b8a-449a-a65d-5dadf7f4483b\") "
Oct 30 05:03:12.233992 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.233974    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476eab0b-9b8a-449a-a65d-5dadf7f4483b-kube-api-access-wpwrd\" (UniqueName: \"kubernetes.io/projected/476eab0b-9b8a-449a-a65d-5dadf7f4483b-kube-api-access-wpwrd\") pod \"476eab0b-9b8a-449a-a65d-5dadf7f4483b\" (UID: \"476eab0b-9b8a-449a-a65d-5dadf7f4483b\") "
Oct 30 05:03:12.234273 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.234249    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476eab0b-9b8a-449a-a65d-5dadf7f4483b-plugins-config-volume" pod "476eab0b-9b8a-449a-a65d-5dadf7f4483b" (UID: "476eab0b-9b8a-449a-a65d-5dadf7f4483b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:03:12.234454 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.234429    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476eab0b-9b8a-449a-a65d-5dadf7f4483b-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:12.236473 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.236451    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476eab0b-9b8a-449a-a65d-5dadf7f4483b-kube-api-access-wpwrd" pod "476eab0b-9b8a-449a-a65d-5dadf7f4483b" (UID: "476eab0b-9b8a-449a-a65d-5dadf7f4483b"). InnerVolumeSpecName "kube-api-access-wpwrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:12.286196 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.285104    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"
Oct 30 05:03:12.335059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.335020    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-plugins-config-volume\") pod \"2bf7fdb4-1e6b-4575-a470-e659bd8306ed\" (UID: \"2bf7fdb4-1e6b-4575-a470-e659bd8306ed\") "
Oct 30 05:03:12.335255 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.335165    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-kube-api-access-fnx74\" (UniqueName: \"kubernetes.io/projected/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-kube-api-access-fnx74\") pod \"2bf7fdb4-1e6b-4575-a470-e659bd8306ed\" (UID: \"2bf7fdb4-1e6b-4575-a470-e659bd8306ed\") "
Oct 30 05:03:12.335478 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.335451    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-plugins-config-volume" pod "2bf7fdb4-1e6b-4575-a470-e659bd8306ed" (UID: "2bf7fdb4-1e6b-4575-a470-e659bd8306ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:03:12.335692 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.335671    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpwrd\" (UniqueName: \"kubernetes.io/projected/476eab0b-9b8a-449a-a65d-5dadf7f4483b-kube-api-access-wpwrd\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:12.335768 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.335697    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:12.337728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.337703    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-kube-api-access-fnx74" pod "2bf7fdb4-1e6b-4575-a470-e659bd8306ed" (UID: "2bf7fdb4-1e6b-4575-a470-e659bd8306ed"). InnerVolumeSpecName "kube-api-access-fnx74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:12.437450 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:12.437418    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnx74\" (UniqueName: \"kubernetes.io/projected/2bf7fdb4-1e6b-4575-a470-e659bd8306ed-kube-api-access-fnx74\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:13.137461 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:13.137430    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"
Oct 30 05:03:13.137904 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:13.137466    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc" event={"ID":"476eab0b-9b8a-449a-a65d-5dadf7f4483b","Type":"ContainerDied","Data":"2e7bc915c68c80101a7f1d5fe4a64da5ea83dd30d1fab04b7acbeb03aec9c230"}
Oct 30 05:03:13.137904 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:13.137532    5072 scope.go:122] "RemoveContainer" containerID="79419a5814aa013c46bac61324f1fc71cb99c415f641ebc74a66cc1f8abd4ba5"
Oct 30 05:03:13.139210 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:13.139187    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"
Oct 30 05:03:13.139379 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:13.139228    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt" event={"ID":"2bf7fdb4-1e6b-4575-a470-e659bd8306ed","Type":"ContainerDied","Data":"6ed6d40cf1729e97b77ea3dce6dcd579bd85f6f20cb0d043722fd1f52d473081"}
Oct 30 05:03:13.149511 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:13.149499    5072 scope.go:122] "RemoveContainer" containerID="c0990b8667af174f479e81ee134f54c93276fcaf1af96f7c70b3fb7609653210"
Oct 30 05:03:13.193228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:13.193203    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"]
Oct 30 05:03:13.205702 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:13.205678    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f9cd17-755544d75d-459sc"]
Oct 30 05:03:13.241429 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:13.241407    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"]
Oct 30 05:03:13.246941 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:13.246919    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f9cd17-85664cc555-9cgvt"]
Oct 30 05:03:14.412981 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:14.412948    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf7fdb4-1e6b-4575-a470-e659bd8306ed" path="/var/lib/kubelet/pods/2bf7fdb4-1e6b-4575-a470-e659bd8306ed/volumes"
Oct 30 05:03:14.413505 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:14.413489    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476eab0b-9b8a-449a-a65d-5dadf7f4483b" path="/var/lib/kubelet/pods/476eab0b-9b8a-449a-a65d-5dadf7f4483b/volumes"
Oct 30 05:03:21.245399 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.245366    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"]
Oct 30 05:03:21.246188 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.246171    5072 cpu_manager.go:414] "containerMap: removing container" podUID="66af17d7-50b8-4277-984a-a5fb20858c56" containerName="payload-pre-processing"
Oct 30 05:03:21.246188 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.246188    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66af17d7-50b8-4277-984a-a5fb20858c56" containerName="payload-pre-processing"
Oct 30 05:03:21.246291 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.246235    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2bf7fdb4-1e6b-4575-a470-e659bd8306ed" containerName="payload-pre-processing"
Oct 30 05:03:21.246291 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.246241    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf7fdb4-1e6b-4575-a470-e659bd8306ed" containerName="payload-pre-processing"
Oct 30 05:03:21.246291 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.246255    5072 cpu_manager.go:414] "containerMap: removing container" podUID="d9d290bd-4133-45ce-9816-a7b94e8d89a4" containerName="istio-proxy"
Oct 30 05:03:21.246291 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.246261    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d290bd-4133-45ce-9816-a7b94e8d89a4" containerName="istio-proxy"
Oct 30 05:03:21.246291 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.246274    5072 cpu_manager.go:414] "containerMap: removing container" podUID="476eab0b-9b8a-449a-a65d-5dadf7f4483b" containerName="payload-processing"
Oct 30 05:03:21.246291 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.246278    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476eab0b-9b8a-449a-a65d-5dadf7f4483b" containerName="payload-processing"
Oct 30 05:03:21.246482 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.246387    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf7fdb4-1e6b-4575-a470-e659bd8306ed" containerName="payload-pre-processing"
Oct 30 05:03:21.246482 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.246398    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d290bd-4133-45ce-9816-a7b94e8d89a4" containerName="istio-proxy"
Oct 30 05:03:21.246482 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.246406    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="476eab0b-9b8a-449a-a65d-5dadf7f4483b" containerName="payload-processing"
Oct 30 05:03:21.246482 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.246415    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="66af17d7-50b8-4277-984a-a5fb20858c56" containerName="payload-pre-processing"
Oct 30 05:03:21.251670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.251649    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:21.268193 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.268169    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"]
Oct 30 05:03:21.275820 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.275797    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:03:21.275991 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.275887    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:03:21.358047 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.358016    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-029657f7-r5vkn\" (UID: \"0a18c910-bf54-4700-8581-4787b9e5372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:21.358251 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.358113    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mmm\" (UniqueName: \"kubernetes.io/projected/0a18c910-bf54-4700-8581-4787b9e5372b-kube-api-access-z5mmm\") pod \"maas-api-revoke-sub-029657f7-r5vkn\" (UID: \"0a18c910-bf54-4700-8581-4787b9e5372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:21.358347 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.358322    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-029657f7-r5vkn\" (UID: \"0a18c910-bf54-4700-8581-4787b9e5372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:21.460034 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.459996    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-029657f7-r5vkn\" (UID: \"0a18c910-bf54-4700-8581-4787b9e5372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:21.460270 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.460060    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-029657f7-r5vkn\" (UID: \"0a18c910-bf54-4700-8581-4787b9e5372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:21.460270 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.460170    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mmm\" (UniqueName: \"kubernetes.io/projected/0a18c910-bf54-4700-8581-4787b9e5372b-kube-api-access-z5mmm\") pod \"maas-api-revoke-sub-029657f7-r5vkn\" (UID: \"0a18c910-bf54-4700-8581-4787b9e5372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:21.460819 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.460791    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-029657f7-r5vkn\" (UID: \"0a18c910-bf54-4700-8581-4787b9e5372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:21.462659 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.462635    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-029657f7-r5vkn\" (UID: \"0a18c910-bf54-4700-8581-4787b9e5372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:21.504284 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.504208    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mmm\" (UniqueName: \"kubernetes.io/projected/0a18c910-bf54-4700-8581-4787b9e5372b-kube-api-access-z5mmm\") pod \"maas-api-revoke-sub-029657f7-r5vkn\" (UID: \"0a18c910-bf54-4700-8581-4787b9e5372b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:21.566910 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.566873    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:21.877403 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:21.877373    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"]
Oct 30 05:03:21.878016 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:03:21.877994    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a18c910_bf54_4700_8581_4787b9e5372b.slice/crio-83c24278b01242958049f6ad2c6b85943d854287cab8f58db9f1d7db6bfc8204 WatchSource:0}: Error finding container 83c24278b01242958049f6ad2c6b85943d854287cab8f58db9f1d7db6bfc8204: Status 404 returned error can't find the container with id 83c24278b01242958049f6ad2c6b85943d854287cab8f58db9f1d7db6bfc8204
Oct 30 05:03:22.213016 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:22.212974    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn" event={"ID":"0a18c910-bf54-4700-8581-4787b9e5372b","Type":"ContainerStarted","Data":"83c24278b01242958049f6ad2c6b85943d854287cab8f58db9f1d7db6bfc8204"}
Oct 30 05:03:23.300105 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:23.299461    5072 generic.go:356] "Generic (PLEG): container finished" podID="0a18c910-bf54-4700-8581-4787b9e5372b" containerID="cc16255f3fca9d7f5ee322091c7a850427d2e81cdf86971dae689131e65ffa6d" exitCode=0
Oct 30 05:03:23.300105 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:23.299565    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn" event={"ID":"0a18c910-bf54-4700-8581-4787b9e5372b","Type":"ContainerDied","Data":"cc16255f3fca9d7f5ee322091c7a850427d2e81cdf86971dae689131e65ffa6d"}
Oct 30 05:03:24.468251 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:24.468227    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:24.614012 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:24.613934    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-xks-ca\") pod \"0a18c910-bf54-4700-8581-4787b9e5372b\" (UID: \"0a18c910-bf54-4700-8581-4787b9e5372b\") "
Oct 30 05:03:24.614241 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:24.614055    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-openshift-ca\") pod \"0a18c910-bf54-4700-8581-4787b9e5372b\" (UID: \"0a18c910-bf54-4700-8581-4787b9e5372b\") "
Oct 30 05:03:24.614241 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:24.614185    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a18c910-bf54-4700-8581-4787b9e5372b-kube-api-access-z5mmm\" (UniqueName: \"kubernetes.io/projected/0a18c910-bf54-4700-8581-4787b9e5372b-kube-api-access-z5mmm\") pod \"0a18c910-bf54-4700-8581-4787b9e5372b\" (UID: \"0a18c910-bf54-4700-8581-4787b9e5372b\") "
Oct 30 05:03:24.614437 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:24.614413    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-openshift-ca" pod "0a18c910-bf54-4700-8581-4787b9e5372b" (UID: "0a18c910-bf54-4700-8581-4787b9e5372b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:03:24.614690 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:24.614676    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:24.616376 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:24.616346    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a18c910-bf54-4700-8581-4787b9e5372b-kube-api-access-z5mmm" pod "0a18c910-bf54-4700-8581-4787b9e5372b" (UID: "0a18c910-bf54-4700-8581-4787b9e5372b"). InnerVolumeSpecName "kube-api-access-z5mmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:24.616485 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:24.616392    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-xks-ca" pod "0a18c910-bf54-4700-8581-4787b9e5372b" (UID: "0a18c910-bf54-4700-8581-4787b9e5372b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:03:24.716088 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:24.716051    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a18c910-bf54-4700-8581-4787b9e5372b-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:24.716207 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:24.716107    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5mmm\" (UniqueName: \"kubernetes.io/projected/0a18c910-bf54-4700-8581-4787b9e5372b-kube-api-access-z5mmm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:25.320338 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:25.320295    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn" event={"ID":"0a18c910-bf54-4700-8581-4787b9e5372b","Type":"ContainerDied","Data":"83c24278b01242958049f6ad2c6b85943d854287cab8f58db9f1d7db6bfc8204"}
Oct 30 05:03:25.320567 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:25.320404    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83c24278b01242958049f6ad2c6b85943d854287cab8f58db9f1d7db6bfc8204"
Oct 30 05:03:25.320567 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:25.320520    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"
Oct 30 05:03:32.647068 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.647036    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"]
Oct 30 05:03:32.648321 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.648297    5072 cpu_manager.go:414] "containerMap: removing container" podUID="0a18c910-bf54-4700-8581-4787b9e5372b" containerName="revoke-keys"
Oct 30 05:03:32.648543 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.648527    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a18c910-bf54-4700-8581-4787b9e5372b" containerName="revoke-keys"
Oct 30 05:03:32.649054 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.649037    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a18c910-bf54-4700-8581-4787b9e5372b" containerName="revoke-keys"
Oct 30 05:03:32.653733 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.653710    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:32.676867 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.676832    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"]
Oct 30 05:03:32.677096 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.677056    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:03:32.677199 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.676990    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:03:32.745657 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.745621    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzskh\" (UniqueName: \"kubernetes.io/projected/79c25813-b497-4d0b-b77d-f9c0662b610d-kube-api-access-nzskh\") pod \"maas-api-revoke-sub-dbaef71c-b2rmk\" (UID: \"79c25813-b497-4d0b-b77d-f9c0662b610d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:32.745864 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.745745    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbaef71c-b2rmk\" (UID: \"79c25813-b497-4d0b-b77d-f9c0662b610d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:32.745864 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.745796    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbaef71c-b2rmk\" (UID: \"79c25813-b497-4d0b-b77d-f9c0662b610d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:32.846729 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.846695    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzskh\" (UniqueName: \"kubernetes.io/projected/79c25813-b497-4d0b-b77d-f9c0662b610d-kube-api-access-nzskh\") pod \"maas-api-revoke-sub-dbaef71c-b2rmk\" (UID: \"79c25813-b497-4d0b-b77d-f9c0662b610d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:32.846916 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.846801    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbaef71c-b2rmk\" (UID: \"79c25813-b497-4d0b-b77d-f9c0662b610d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:32.846916 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.846849    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbaef71c-b2rmk\" (UID: \"79c25813-b497-4d0b-b77d-f9c0662b610d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:32.847589 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.847561    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbaef71c-b2rmk\" (UID: \"79c25813-b497-4d0b-b77d-f9c0662b610d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:32.849503 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.849484    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbaef71c-b2rmk\" (UID: \"79c25813-b497-4d0b-b77d-f9c0662b610d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:32.870534 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.870514    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzskh\" (UniqueName: \"kubernetes.io/projected/79c25813-b497-4d0b-b77d-f9c0662b610d-kube-api-access-nzskh\") pod \"maas-api-revoke-sub-dbaef71c-b2rmk\" (UID: \"79c25813-b497-4d0b-b77d-f9c0662b610d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:32.984997 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:32.984972    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:33.292664 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:33.292636    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"]
Oct 30 05:03:33.292810 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:03:33.292788    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c25813_b497_4d0b_b77d_f9c0662b610d.slice/crio-adec5ad454fddfea84599cd3494fc3c86f008eb976240c7f018420ee152a7457 WatchSource:0}: Error finding container adec5ad454fddfea84599cd3494fc3c86f008eb976240c7f018420ee152a7457: Status 404 returned error can't find the container with id adec5ad454fddfea84599cd3494fc3c86f008eb976240c7f018420ee152a7457
Oct 30 05:03:33.385014 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:33.384982    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk" event={"ID":"79c25813-b497-4d0b-b77d-f9c0662b610d","Type":"ContainerStarted","Data":"adec5ad454fddfea84599cd3494fc3c86f008eb976240c7f018420ee152a7457"}
Oct 30 05:03:34.395069 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:34.395033    5072 generic.go:356] "Generic (PLEG): container finished" podID="79c25813-b497-4d0b-b77d-f9c0662b610d" containerID="d98ec538ac69ef288d2d297cde94430e5468758ec4e27cb804c17e1ef44cdb11" exitCode=0
Oct 30 05:03:34.395475 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:34.395117    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk" event={"ID":"79c25813-b497-4d0b-b77d-f9c0662b610d","Type":"ContainerDied","Data":"d98ec538ac69ef288d2d297cde94430e5468758ec4e27cb804c17e1ef44cdb11"}
Oct 30 05:03:35.584630 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:35.584597    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:35.687998 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:35.687970    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-openshift-ca\") pod \"79c25813-b497-4d0b-b77d-f9c0662b610d\" (UID: \"79c25813-b497-4d0b-b77d-f9c0662b610d\") "
Oct 30 05:03:35.688151 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:35.688070    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-xks-ca\") pod \"79c25813-b497-4d0b-b77d-f9c0662b610d\" (UID: \"79c25813-b497-4d0b-b77d-f9c0662b610d\") "
Oct 30 05:03:35.688207 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:35.688156    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c25813-b497-4d0b-b77d-f9c0662b610d-kube-api-access-nzskh\" (UniqueName: \"kubernetes.io/projected/79c25813-b497-4d0b-b77d-f9c0662b610d-kube-api-access-nzskh\") pod \"79c25813-b497-4d0b-b77d-f9c0662b610d\" (UID: \"79c25813-b497-4d0b-b77d-f9c0662b610d\") "
Oct 30 05:03:35.688474 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:35.688448    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-openshift-ca" pod "79c25813-b497-4d0b-b77d-f9c0662b610d" (UID: "79c25813-b497-4d0b-b77d-f9c0662b610d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:03:35.688703 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:35.688685    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:35.690458 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:35.690428    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-xks-ca" pod "79c25813-b497-4d0b-b77d-f9c0662b610d" (UID: "79c25813-b497-4d0b-b77d-f9c0662b610d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:03:35.690564 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:35.690487    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c25813-b497-4d0b-b77d-f9c0662b610d-kube-api-access-nzskh" pod "79c25813-b497-4d0b-b77d-f9c0662b610d" (UID: "79c25813-b497-4d0b-b77d-f9c0662b610d"). InnerVolumeSpecName "kube-api-access-nzskh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:35.789686 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:35.789655    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c25813-b497-4d0b-b77d-f9c0662b610d-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:35.789686 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:35.789681    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzskh\" (UniqueName: \"kubernetes.io/projected/79c25813-b497-4d0b-b77d-f9c0662b610d-kube-api-access-nzskh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:36.416406 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:36.416376    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"
Oct 30 05:03:36.416406 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:36.416384    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk" event={"ID":"79c25813-b497-4d0b-b77d-f9c0662b610d","Type":"ContainerDied","Data":"adec5ad454fddfea84599cd3494fc3c86f008eb976240c7f018420ee152a7457"}
Oct 30 05:03:36.416655 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:36.416424    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adec5ad454fddfea84599cd3494fc3c86f008eb976240c7f018420ee152a7457"
Oct 30 05:03:41.753644 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:03:41.753559    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:03:46.466613 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:46.466559    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"]
Oct 30 05:03:46.467157 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:46.466842    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy" containerID="cri-o://0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112" gracePeriod=30
Oct 30 05:03:48.557854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:48.557809    5072 patch_prober.go:29] interesting pod/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:03:48.558266 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:48.557878    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:03:51.778483 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.778451    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 05:03:51.844349 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.844319    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-data\") pod \"bfa3c63a-eaf0-4aec-b463-adc66812e632\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") "
Oct 30 05:03:51.844524 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.844429    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-podinfo\") pod \"bfa3c63a-eaf0-4aec-b463-adc66812e632\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") "
Oct 30 05:03:51.844524 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.844482    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-kube-api-access-hpnxj\" (UniqueName: \"kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-kube-api-access-hpnxj\") pod \"bfa3c63a-eaf0-4aec-b463-adc66812e632\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") "
Oct 30 05:03:51.844656 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.844521    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-credential-socket\") pod \"bfa3c63a-eaf0-4aec-b463-adc66812e632\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") "
Oct 30 05:03:51.844656 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.844548    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-token\" (UniqueName: \"kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-token\") pod \"bfa3c63a-eaf0-4aec-b463-adc66812e632\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") "
Oct 30 05:03:51.844656 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.844630    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-data" pod "bfa3c63a-eaf0-4aec-b463-adc66812e632" (UID: "bfa3c63a-eaf0-4aec-b463-adc66812e632"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:03:51.844826 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.844669    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-certs\") pod \"bfa3c63a-eaf0-4aec-b463-adc66812e632\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") "
Oct 30 05:03:51.844826 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.844746    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-envoy\") pod \"bfa3c63a-eaf0-4aec-b463-adc66812e632\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") "
Oct 30 05:03:51.844826 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.844792    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-socket\") pod \"bfa3c63a-eaf0-4aec-b463-adc66812e632\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") "
Oct 30 05:03:51.845001 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.844862    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa3c63a-eaf0-4aec-b463-adc66812e632-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa3c63a-eaf0-4aec-b463-adc66812e632-istiod-ca-cert\") pod \"bfa3c63a-eaf0-4aec-b463-adc66812e632\" (UID: \"bfa3c63a-eaf0-4aec-b463-adc66812e632\") "
Oct 30 05:03:51.845001 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.844941    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-certs" pod "bfa3c63a-eaf0-4aec-b463-adc66812e632" (UID: "bfa3c63a-eaf0-4aec-b463-adc66812e632"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:03:51.845130 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.845063    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-socket" pod "bfa3c63a-eaf0-4aec-b463-adc66812e632" (UID: "bfa3c63a-eaf0-4aec-b463-adc66812e632"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:03:51.845177 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.845130    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-credential-socket" pod "bfa3c63a-eaf0-4aec-b463-adc66812e632" (UID: "bfa3c63a-eaf0-4aec-b463-adc66812e632"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:03:51.845341 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.845316    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa3c63a-eaf0-4aec-b463-adc66812e632-istiod-ca-cert" pod "bfa3c63a-eaf0-4aec-b463-adc66812e632" (UID: "bfa3c63a-eaf0-4aec-b463-adc66812e632"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:03:51.845613 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.845576    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:51.845613 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.845598    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa3c63a-eaf0-4aec-b463-adc66812e632-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:51.845613 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.845611    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:51.845787 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.845622    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:51.845787 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.845634    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:51.847414 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.847372    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-podinfo" pod "bfa3c63a-eaf0-4aec-b463-adc66812e632" (UID: "bfa3c63a-eaf0-4aec-b463-adc66812e632"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:03:51.847541 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.847491    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-kube-api-access-hpnxj" pod "bfa3c63a-eaf0-4aec-b463-adc66812e632" (UID: "bfa3c63a-eaf0-4aec-b463-adc66812e632"). InnerVolumeSpecName "kube-api-access-hpnxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:51.847958 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.847934    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-envoy" pod "bfa3c63a-eaf0-4aec-b463-adc66812e632" (UID: "bfa3c63a-eaf0-4aec-b463-adc66812e632"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:03:51.848120 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.847992    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-token" pod "bfa3c63a-eaf0-4aec-b463-adc66812e632" (UID: "bfa3c63a-eaf0-4aec-b463-adc66812e632"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:51.947218 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.947113    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:51.947218 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.947157    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpnxj\" (UniqueName: \"kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-kube-api-access-hpnxj\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:51.947218 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.947172    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:51.947218 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:51.947186    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa3c63a-eaf0-4aec-b463-adc66812e632-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:03:52.561911 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:52.561874    5072 generic.go:356] "Generic (PLEG): container finished" podID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerID="0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112" exitCode=0
Oct 30 05:03:52.562150 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:52.561935    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"
Oct 30 05:03:52.562150 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:52.561958    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" event={"ID":"bfa3c63a-eaf0-4aec-b463-adc66812e632","Type":"ContainerDied","Data":"0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112"}
Oct 30 05:03:52.562150 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:52.562015    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75" event={"ID":"bfa3c63a-eaf0-4aec-b463-adc66812e632","Type":"ContainerDied","Data":"c7a9036065d57da0de4a7dc75b853819789cacc2b707d23376273f49e4bb1436"}
Oct 30 05:03:52.562150 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:52.562041    5072 scope.go:122] "RemoveContainer" containerID="0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112"
Oct 30 05:03:52.574350 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:52.574330    5072 scope.go:122] "RemoveContainer" containerID="6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd"
Oct 30 05:03:52.590832 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:52.590806    5072 scope.go:122] "RemoveContainer" containerID="0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112"
Oct 30 05:03:52.591289 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:03:52.591146    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112\": container with ID starting with 0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112 not found: ID does not exist" containerID="0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112"
Oct 30 05:03:52.591289 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:52.591185    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112"} err="failed to get container status \"0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112\": rpc error: code = NotFound desc = could not find container \"0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112\": container with ID starting with 0ba3502584d3b7f4f9ca2989f8a211777efbd860810389807eef8d64e71a5112 not found: ID does not exist"
Oct 30 05:03:52.591289 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:52.591211    5072 scope.go:122] "RemoveContainer" containerID="6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd"
Oct 30 05:03:52.591527 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:03:52.591505    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd\": container with ID starting with 6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd not found: ID does not exist" containerID="6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd"
Oct 30 05:03:52.591578 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:52.591537    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd"} err="failed to get container status \"6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd\": rpc error: code = NotFound desc = could not find container \"6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd\": container with ID starting with 6759b624c366915d8f6b5fd60bba76e7f3e8a28d126f0296d5b577c0f3da8efd not found: ID does not exist"
Oct 30 05:03:52.649030 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:52.648994    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"]
Oct 30 05:03:52.673775 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:52.673687    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f9cd17-openshift-default-58dbc45b7c-fhj75"]
Oct 30 05:03:54.409886 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:54.409850    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" path="/var/lib/kubelet/pods/bfa3c63a-eaf0-4aec-b463-adc66812e632/volumes"
Oct 30 05:03:58.268623 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:58.268589    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"]
Oct 30 05:03:58.337672 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:03:58.337640    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"]
Oct 30 05:04:03.269252 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.269202    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5" podUID="e9dc2b2c-76e1-4433-962c-179a352a8e8f" containerName="payload-processing" containerID="cri-o://cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541" gracePeriod=25
Oct 30 05:04:03.338729 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.338689    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7" podUID="a562af18-3e2a-493a-838d-0966f513c9c2" containerName="payload-pre-processing" containerID="cri-o://8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df" gracePeriod=25
Oct 30 05:04:03.566793 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.566764    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"
Oct 30 05:04:03.673597 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.673557    5072 generic.go:356] "Generic (PLEG): container finished" podID="e9dc2b2c-76e1-4433-962c-179a352a8e8f" containerID="cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541" exitCode=0
Oct 30 05:04:03.673812 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.673613    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"
Oct 30 05:04:03.673812 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.673645    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5" event={"ID":"e9dc2b2c-76e1-4433-962c-179a352a8e8f","Type":"ContainerDied","Data":"cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541"}
Oct 30 05:04:03.673812 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.673699    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5" event={"ID":"e9dc2b2c-76e1-4433-962c-179a352a8e8f","Type":"ContainerDied","Data":"2727b7c02d63f9549b5f048e6406c9e93aa58bb8033795a5f77bdbfe7cbdbc06"}
Oct 30 05:04:03.673812 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.673725    5072 scope.go:122] "RemoveContainer" containerID="cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541"
Oct 30 05:04:03.685187 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.685164    5072 scope.go:122] "RemoveContainer" containerID="cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541"
Oct 30 05:04:03.685432 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:04:03.685412    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541\": container with ID starting with cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541 not found: ID does not exist" containerID="cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541"
Oct 30 05:04:03.685489 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.685439    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541"} err="failed to get container status \"cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541\": rpc error: code = NotFound desc = could not find container \"cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541\": container with ID starting with cce85b47252711399646090c2cfbfb79ceaa0c6dea032b6c1ded49b0293e6541 not found: ID does not exist"
Oct 30 05:04:03.722120 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.722053    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9dc2b2c-76e1-4433-962c-179a352a8e8f-kube-api-access-rvz79\" (UniqueName: \"kubernetes.io/projected/e9dc2b2c-76e1-4433-962c-179a352a8e8f-kube-api-access-rvz79\") pod \"e9dc2b2c-76e1-4433-962c-179a352a8e8f\" (UID: \"e9dc2b2c-76e1-4433-962c-179a352a8e8f\") "
Oct 30 05:04:03.722384 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.722363    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9dc2b2c-76e1-4433-962c-179a352a8e8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dc2b2c-76e1-4433-962c-179a352a8e8f-plugins-config-volume\") pod \"e9dc2b2c-76e1-4433-962c-179a352a8e8f\" (UID: \"e9dc2b2c-76e1-4433-962c-179a352a8e8f\") "
Oct 30 05:04:03.722728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.722709    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9dc2b2c-76e1-4433-962c-179a352a8e8f-plugins-config-volume" pod "e9dc2b2c-76e1-4433-962c-179a352a8e8f" (UID: "e9dc2b2c-76e1-4433-962c-179a352a8e8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:04:03.722893 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.722875    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dc2b2c-76e1-4433-962c-179a352a8e8f-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:04:03.724355 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.724332    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9dc2b2c-76e1-4433-962c-179a352a8e8f-kube-api-access-rvz79" pod "e9dc2b2c-76e1-4433-962c-179a352a8e8f" (UID: "e9dc2b2c-76e1-4433-962c-179a352a8e8f"). InnerVolumeSpecName "kube-api-access-rvz79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:04:03.824227 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:03.824149    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvz79\" (UniqueName: \"kubernetes.io/projected/e9dc2b2c-76e1-4433-962c-179a352a8e8f-kube-api-access-rvz79\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:04:04.025566 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:04.025528    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"]
Oct 30 05:04:04.051028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:04.050998    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-449d83-6f7f8b8c6-h7dc5"]
Oct 30 05:04:04.410636 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:04.410603    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9dc2b2c-76e1-4433-962c-179a352a8e8f" path="/var/lib/kubelet/pods/e9dc2b2c-76e1-4433-962c-179a352a8e8f/volumes"
Oct 30 05:04:11.777099 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:04:11.776398    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:04:12.304554 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:12.304500    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7" podUID="a562af18-3e2a-493a-838d-0966f513c9c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:04:22.304372 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:22.304330    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7" podUID="a562af18-3e2a-493a-838d-0966f513c9c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:04:23.626740 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.626710    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"
Oct 30 05:04:23.806693 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.806659    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a562af18-3e2a-493a-838d-0966f513c9c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a562af18-3e2a-493a-838d-0966f513c9c2-plugins-config-volume\") pod \"a562af18-3e2a-493a-838d-0966f513c9c2\" (UID: \"a562af18-3e2a-493a-838d-0966f513c9c2\") "
Oct 30 05:04:23.806865 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.806794    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a562af18-3e2a-493a-838d-0966f513c9c2-kube-api-access-qjzgh\" (UniqueName: \"kubernetes.io/projected/a562af18-3e2a-493a-838d-0966f513c9c2-kube-api-access-qjzgh\") pod \"a562af18-3e2a-493a-838d-0966f513c9c2\" (UID: \"a562af18-3e2a-493a-838d-0966f513c9c2\") "
Oct 30 05:04:23.807146 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.807119    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a562af18-3e2a-493a-838d-0966f513c9c2-plugins-config-volume" pod "a562af18-3e2a-493a-838d-0966f513c9c2" (UID: "a562af18-3e2a-493a-838d-0966f513c9c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:04:23.807269 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.807254    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a562af18-3e2a-493a-838d-0966f513c9c2-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:04:23.809052 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.809033    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a562af18-3e2a-493a-838d-0966f513c9c2-kube-api-access-qjzgh" pod "a562af18-3e2a-493a-838d-0966f513c9c2" (UID: "a562af18-3e2a-493a-838d-0966f513c9c2"). InnerVolumeSpecName "kube-api-access-qjzgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:04:23.901697 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.901632    5072 generic.go:356] "Generic (PLEG): container finished" podID="a562af18-3e2a-493a-838d-0966f513c9c2" containerID="8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df" exitCode=0
Oct 30 05:04:23.901697 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.901675    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"
Oct 30 05:04:23.901837 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.901717    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7" event={"ID":"a562af18-3e2a-493a-838d-0966f513c9c2","Type":"ContainerDied","Data":"8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df"}
Oct 30 05:04:23.901837 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.901770    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7" event={"ID":"a562af18-3e2a-493a-838d-0966f513c9c2","Type":"ContainerDied","Data":"2a7dabf3cf7eeb41ccc761cf1ec0e816e962ad49d9645e0e8a06c4a479e27083"}
Oct 30 05:04:23.901837 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.901794    5072 scope.go:122] "RemoveContainer" containerID="8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df"
Oct 30 05:04:23.908483 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.908458    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjzgh\" (UniqueName: \"kubernetes.io/projected/a562af18-3e2a-493a-838d-0966f513c9c2-kube-api-access-qjzgh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:04:23.913222 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.913193    5072 scope.go:122] "RemoveContainer" containerID="8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df"
Oct 30 05:04:23.913494 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:04:23.913478    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df\": container with ID starting with 8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df not found: ID does not exist" containerID="8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df"
Oct 30 05:04:23.913539 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.913502    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df"} err="failed to get container status \"8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df\": rpc error: code = NotFound desc = could not find container \"8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df\": container with ID starting with 8e2b3dadacbc6ae83766636bdb773f7b2a194701390cf1083223aadadb1844df not found: ID does not exist"
Oct 30 05:04:23.957557 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.957536    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"]
Oct 30 05:04:23.973441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:23.973419    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-449d83-58478f6975-j7hc7"]
Oct 30 05:04:24.411794 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:24.411754    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a562af18-3e2a-493a-838d-0966f513c9c2" path="/var/lib/kubelet/pods/a562af18-3e2a-493a-838d-0966f513c9c2/volumes"
Oct 30 05:04:38.108801 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.108762    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"]
Oct 30 05:04:38.109785 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.109761    5072 cpu_manager.go:414] "containerMap: removing container" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy"
Oct 30 05:04:38.109910 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.109785    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy"
Oct 30 05:04:38.109910 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.109802    5072 cpu_manager.go:414] "containerMap: removing container" podUID="79c25813-b497-4d0b-b77d-f9c0662b610d" containerName="revoke-keys"
Oct 30 05:04:38.109910 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.109809    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c25813-b497-4d0b-b77d-f9c0662b610d" containerName="revoke-keys"
Oct 30 05:04:38.109910 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.109829    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a562af18-3e2a-493a-838d-0966f513c9c2" containerName="payload-pre-processing"
Oct 30 05:04:38.109910 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.109837    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a562af18-3e2a-493a-838d-0966f513c9c2" containerName="payload-pre-processing"
Oct 30 05:04:38.110252 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.109926    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e9dc2b2c-76e1-4433-962c-179a352a8e8f" containerName="payload-processing"
Oct 30 05:04:38.110252 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.109935    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9dc2b2c-76e1-4433-962c-179a352a8e8f" containerName="payload-processing"
Oct 30 05:04:38.110252 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.110127    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy"
Oct 30 05:04:38.110252 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.110152    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9dc2b2c-76e1-4433-962c-179a352a8e8f" containerName="payload-processing"
Oct 30 05:04:38.110252 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.110166    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c25813-b497-4d0b-b77d-f9c0662b610d" containerName="revoke-keys"
Oct 30 05:04:38.110252 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.110175    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a562af18-3e2a-493a-838d-0966f513c9c2" containerName="payload-pre-processing"
Oct 30 05:04:38.120498 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.120471    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:38.132183 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.132161    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"]
Oct 30 05:04:38.139308 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.139265    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:04:38.139443 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.139309    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:04:38.195863 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.195824    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds5kk\" (UniqueName: \"kubernetes.io/projected/385b9725-5079-42e7-87b0-d361776db342-kube-api-access-ds5kk\") pod \"maas-api-revoke-sub-3907dfa9-pkslz\" (UID: \"385b9725-5079-42e7-87b0-d361776db342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:38.196037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.195888    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385b9725-5079-42e7-87b0-d361776db342-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3907dfa9-pkslz\" (UID: \"385b9725-5079-42e7-87b0-d361776db342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:38.196135 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.196113    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385b9725-5079-42e7-87b0-d361776db342-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3907dfa9-pkslz\" (UID: \"385b9725-5079-42e7-87b0-d361776db342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:38.297026 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.296994    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds5kk\" (UniqueName: \"kubernetes.io/projected/385b9725-5079-42e7-87b0-d361776db342-kube-api-access-ds5kk\") pod \"maas-api-revoke-sub-3907dfa9-pkslz\" (UID: \"385b9725-5079-42e7-87b0-d361776db342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:38.297236 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.297092    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385b9725-5079-42e7-87b0-d361776db342-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3907dfa9-pkslz\" (UID: \"385b9725-5079-42e7-87b0-d361776db342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:38.297236 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.297223    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385b9725-5079-42e7-87b0-d361776db342-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3907dfa9-pkslz\" (UID: \"385b9725-5079-42e7-87b0-d361776db342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:38.297892 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.297871    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385b9725-5079-42e7-87b0-d361776db342-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3907dfa9-pkslz\" (UID: \"385b9725-5079-42e7-87b0-d361776db342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:38.299842 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.299805    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385b9725-5079-42e7-87b0-d361776db342-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3907dfa9-pkslz\" (UID: \"385b9725-5079-42e7-87b0-d361776db342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:38.321142 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.321107    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds5kk\" (UniqueName: \"kubernetes.io/projected/385b9725-5079-42e7-87b0-d361776db342-kube-api-access-ds5kk\") pod \"maas-api-revoke-sub-3907dfa9-pkslz\" (UID: \"385b9725-5079-42e7-87b0-d361776db342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:38.439602 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.439573    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:38.770270 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:04:38.770236    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385b9725_5079_42e7_87b0_d361776db342.slice/crio-10edd9f51c5ccd0819bb3320b83a0aebac2e90eaf7b6f08f6c7cbef708e73671 WatchSource:0}: Error finding container 10edd9f51c5ccd0819bb3320b83a0aebac2e90eaf7b6f08f6c7cbef708e73671: Status 404 returned error can't find the container with id 10edd9f51c5ccd0819bb3320b83a0aebac2e90eaf7b6f08f6c7cbef708e73671
Oct 30 05:04:38.770405 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:38.770355    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"]
Oct 30 05:04:39.048017 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:39.047920    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz" event={"ID":"385b9725-5079-42e7-87b0-d361776db342","Type":"ContainerStarted","Data":"10edd9f51c5ccd0819bb3320b83a0aebac2e90eaf7b6f08f6c7cbef708e73671"}
Oct 30 05:04:40.058442 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:40.058349    5072 generic.go:356] "Generic (PLEG): container finished" podID="385b9725-5079-42e7-87b0-d361776db342" containerID="2912121b9a92b42a4abc68481925675f1e65d5080ac1c9d1813e434cfda03ada" exitCode=0
Oct 30 05:04:40.058932 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:40.058432    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz" event={"ID":"385b9725-5079-42e7-87b0-d361776db342","Type":"ContainerDied","Data":"2912121b9a92b42a4abc68481925675f1e65d5080ac1c9d1813e434cfda03ada"}
Oct 30 05:04:41.228717 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:41.228690    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:41.340518 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:41.340424    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/385b9725-5079-42e7-87b0-d361776db342-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385b9725-5079-42e7-87b0-d361776db342-maas-api-openshift-ca\") pod \"385b9725-5079-42e7-87b0-d361776db342\" (UID: \"385b9725-5079-42e7-87b0-d361776db342\") "
Oct 30 05:04:41.340681 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:41.340563    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385b9725-5079-42e7-87b0-d361776db342-kube-api-access-ds5kk\" (UniqueName: \"kubernetes.io/projected/385b9725-5079-42e7-87b0-d361776db342-kube-api-access-ds5kk\") pod \"385b9725-5079-42e7-87b0-d361776db342\" (UID: \"385b9725-5079-42e7-87b0-d361776db342\") "
Oct 30 05:04:41.340681 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:41.340586    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/385b9725-5079-42e7-87b0-d361776db342-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385b9725-5079-42e7-87b0-d361776db342-maas-api-xks-ca\") pod \"385b9725-5079-42e7-87b0-d361776db342\" (UID: \"385b9725-5079-42e7-87b0-d361776db342\") "
Oct 30 05:04:41.340887 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:41.340860    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385b9725-5079-42e7-87b0-d361776db342-maas-api-openshift-ca" pod "385b9725-5079-42e7-87b0-d361776db342" (UID: "385b9725-5079-42e7-87b0-d361776db342"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:04:41.341112 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:41.341054    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385b9725-5079-42e7-87b0-d361776db342-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:04:41.342920 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:41.342896    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/385b9725-5079-42e7-87b0-d361776db342-maas-api-xks-ca" pod "385b9725-5079-42e7-87b0-d361776db342" (UID: "385b9725-5079-42e7-87b0-d361776db342"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:04:41.343014 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:41.342954    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385b9725-5079-42e7-87b0-d361776db342-kube-api-access-ds5kk" pod "385b9725-5079-42e7-87b0-d361776db342" (UID: "385b9725-5079-42e7-87b0-d361776db342"). InnerVolumeSpecName "kube-api-access-ds5kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:04:41.442562 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:41.442525    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds5kk\" (UniqueName: \"kubernetes.io/projected/385b9725-5079-42e7-87b0-d361776db342-kube-api-access-ds5kk\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:04:41.442562 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:41.442556    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385b9725-5079-42e7-87b0-d361776db342-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:04:41.754979 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:04:41.754935    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:04:42.080818 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:42.080735    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"
Oct 30 05:04:42.081260 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:42.080795    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz" event={"ID":"385b9725-5079-42e7-87b0-d361776db342","Type":"ContainerDied","Data":"10edd9f51c5ccd0819bb3320b83a0aebac2e90eaf7b6f08f6c7cbef708e73671"}
Oct 30 05:04:42.081391 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:42.081353    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10edd9f51c5ccd0819bb3320b83a0aebac2e90eaf7b6f08f6c7cbef708e73671"
Oct 30 05:04:46.576840 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:46.576808    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"]
Oct 30 05:04:46.611304 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:46.611277    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"]
Oct 30 05:04:51.577676 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:51.577612    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj" podUID="71d6aa4c-661b-4bb8-a051-87ed6f430464" containerName="payload-processing" containerID="cri-o://4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c" gracePeriod=25
Oct 30 05:04:51.612093 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:51.612048    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq" podUID="49ab9ae3-71d5-469d-8860-50240e90aa21" containerName="payload-pre-processing" containerID="cri-o://3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e" gracePeriod=25
Oct 30 05:04:51.873799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:51.873771    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"
Oct 30 05:04:51.911816 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:51.911790    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d6aa4c-661b-4bb8-a051-87ed6f430464-kube-api-access-n56br\" (UniqueName: \"kubernetes.io/projected/71d6aa4c-661b-4bb8-a051-87ed6f430464-kube-api-access-n56br\") pod \"71d6aa4c-661b-4bb8-a051-87ed6f430464\" (UID: \"71d6aa4c-661b-4bb8-a051-87ed6f430464\") "
Oct 30 05:04:51.911977 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:51.911959    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d6aa4c-661b-4bb8-a051-87ed6f430464-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d6aa4c-661b-4bb8-a051-87ed6f430464-plugins-config-volume\") pod \"71d6aa4c-661b-4bb8-a051-87ed6f430464\" (UID: \"71d6aa4c-661b-4bb8-a051-87ed6f430464\") "
Oct 30 05:04:51.912310 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:51.912288    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d6aa4c-661b-4bb8-a051-87ed6f430464-plugins-config-volume" pod "71d6aa4c-661b-4bb8-a051-87ed6f430464" (UID: "71d6aa4c-661b-4bb8-a051-87ed6f430464"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:04:51.912507 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:51.912492    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d6aa4c-661b-4bb8-a051-87ed6f430464-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:04:51.914164 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:51.914138    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d6aa4c-661b-4bb8-a051-87ed6f430464-kube-api-access-n56br" pod "71d6aa4c-661b-4bb8-a051-87ed6f430464" (UID: "71d6aa4c-661b-4bb8-a051-87ed6f430464"). InnerVolumeSpecName "kube-api-access-n56br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:04:52.013490 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:52.013462    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n56br\" (UniqueName: \"kubernetes.io/projected/71d6aa4c-661b-4bb8-a051-87ed6f430464-kube-api-access-n56br\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:04:52.157713 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:52.157633    5072 generic.go:356] "Generic (PLEG): container finished" podID="71d6aa4c-661b-4bb8-a051-87ed6f430464" containerID="4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c" exitCode=0
Oct 30 05:04:52.157713 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:52.157686    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"
Oct 30 05:04:52.157713 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:52.157686    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj" event={"ID":"71d6aa4c-661b-4bb8-a051-87ed6f430464","Type":"ContainerDied","Data":"4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c"}
Oct 30 05:04:52.158023 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:52.157743    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj" event={"ID":"71d6aa4c-661b-4bb8-a051-87ed6f430464","Type":"ContainerDied","Data":"b6aaa302535e393b90337ec3dbe82fec58b6510f8a43910d2604eb695b6f51cf"}
Oct 30 05:04:52.158023 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:52.157766    5072 scope.go:122] "RemoveContainer" containerID="4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c"
Oct 30 05:04:52.195441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:52.195407    5072 scope.go:122] "RemoveContainer" containerID="4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c"
Oct 30 05:04:52.195923 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:04:52.195870    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c\": container with ID starting with 4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c not found: ID does not exist" containerID="4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c"
Oct 30 05:04:52.196070 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:52.195917    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c"} err="failed to get container status \"4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c\": rpc error: code = NotFound desc = could not find container \"4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c\": container with ID starting with 4018fa13f82f3c300831d1cc4256be6691598ed0e2301e996cd28214d136cc2c not found: ID does not exist"
Oct 30 05:04:52.230691 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:52.230659    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"]
Oct 30 05:04:52.242913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:52.242885    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fd27a9-5f8758bbd8-bkggj"]
Oct 30 05:04:52.410228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:52.410161    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d6aa4c-661b-4bb8-a051-87ed6f430464" path="/var/lib/kubelet/pods/71d6aa4c-661b-4bb8-a051-87ed6f430464/volumes"
Oct 30 05:04:59.632039 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:04:59.632000    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq" podUID="49ab9ae3-71d5-469d-8860-50240e90aa21" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:05:09.632124 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:09.632063    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq" podUID="49ab9ae3-71d5-469d-8860-50240e90aa21" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:05:11.755430 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:05:11.755376    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:05:11.895373 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:11.895346    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"
Oct 30 05:05:11.991809 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:11.991775    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ab9ae3-71d5-469d-8860-50240e90aa21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ab9ae3-71d5-469d-8860-50240e90aa21-plugins-config-volume\") pod \"49ab9ae3-71d5-469d-8860-50240e90aa21\" (UID: \"49ab9ae3-71d5-469d-8860-50240e90aa21\") "
Oct 30 05:05:11.992010 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:11.991910    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ab9ae3-71d5-469d-8860-50240e90aa21-kube-api-access-fk69w\" (UniqueName: \"kubernetes.io/projected/49ab9ae3-71d5-469d-8860-50240e90aa21-kube-api-access-fk69w\") pod \"49ab9ae3-71d5-469d-8860-50240e90aa21\" (UID: \"49ab9ae3-71d5-469d-8860-50240e90aa21\") "
Oct 30 05:05:11.992183 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:11.992160    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ab9ae3-71d5-469d-8860-50240e90aa21-plugins-config-volume" pod "49ab9ae3-71d5-469d-8860-50240e90aa21" (UID: "49ab9ae3-71d5-469d-8860-50240e90aa21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:05:11.992518 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:11.992484    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ab9ae3-71d5-469d-8860-50240e90aa21-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:11.994169 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:11.994146    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ab9ae3-71d5-469d-8860-50240e90aa21-kube-api-access-fk69w" pod "49ab9ae3-71d5-469d-8860-50240e90aa21" (UID: "49ab9ae3-71d5-469d-8860-50240e90aa21"). InnerVolumeSpecName "kube-api-access-fk69w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:05:12.093884 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:12.093855    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk69w\" (UniqueName: \"kubernetes.io/projected/49ab9ae3-71d5-469d-8860-50240e90aa21-kube-api-access-fk69w\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:12.327381 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:12.327340    5072 generic.go:356] "Generic (PLEG): container finished" podID="49ab9ae3-71d5-469d-8860-50240e90aa21" containerID="3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e" exitCode=0
Oct 30 05:05:12.327552 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:12.327394    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"
Oct 30 05:05:12.327552 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:12.327415    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq" event={"ID":"49ab9ae3-71d5-469d-8860-50240e90aa21","Type":"ContainerDied","Data":"3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e"}
Oct 30 05:05:12.327552 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:12.327482    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq" event={"ID":"49ab9ae3-71d5-469d-8860-50240e90aa21","Type":"ContainerDied","Data":"7f0dbabdfed023900a460f28280e805a0aac2a624b20925bb7cf9890df392ba3"}
Oct 30 05:05:12.327552 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:12.327513    5072 scope.go:122] "RemoveContainer" containerID="3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e"
Oct 30 05:05:12.339539 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:12.339518    5072 scope.go:122] "RemoveContainer" containerID="3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e"
Oct 30 05:05:12.339807 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:05:12.339790    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e\": container with ID starting with 3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e not found: ID does not exist" containerID="3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e"
Oct 30 05:05:12.339885 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:12.339814    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e"} err="failed to get container status \"3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e\": rpc error: code = NotFound desc = could not find container \"3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e\": container with ID starting with 3e2abbe2dec1c43ced7bf69f68bdc0bdb1cdb8221f77bc4aaf83aca1839dca4e not found: ID does not exist"
Oct 30 05:05:12.382952 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:12.382925    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"]
Oct 30 05:05:12.387021 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:12.386994    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fd27a9-549448bc99-kdsxq"]
Oct 30 05:05:12.410592 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:12.410568    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ab9ae3-71d5-469d-8860-50240e90aa21" path="/var/lib/kubelet/pods/49ab9ae3-71d5-469d-8860-50240e90aa21/volumes"
Oct 30 05:05:20.977243 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.977209    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"]
Oct 30 05:05:20.977979 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.977965    5072 cpu_manager.go:414] "containerMap: removing container" podUID="49ab9ae3-71d5-469d-8860-50240e90aa21" containerName="payload-pre-processing"
Oct 30 05:05:20.978024 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.977980    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ab9ae3-71d5-469d-8860-50240e90aa21" containerName="payload-pre-processing"
Oct 30 05:05:20.978024 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.977993    5072 cpu_manager.go:414] "containerMap: removing container" podUID="71d6aa4c-661b-4bb8-a051-87ed6f430464" containerName="payload-processing"
Oct 30 05:05:20.978024 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.977997    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d6aa4c-661b-4bb8-a051-87ed6f430464" containerName="payload-processing"
Oct 30 05:05:20.978024 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.978013    5072 cpu_manager.go:414] "containerMap: removing container" podUID="385b9725-5079-42e7-87b0-d361776db342" containerName="revoke-keys"
Oct 30 05:05:20.978024 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.978018    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385b9725-5079-42e7-87b0-d361776db342" containerName="revoke-keys"
Oct 30 05:05:20.978024 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.978025    5072 cpu_manager.go:414] "containerMap: removing container" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy"
Oct 30 05:05:20.978292 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.978031    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy"
Oct 30 05:05:20.978292 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.978191    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="385b9725-5079-42e7-87b0-d361776db342" containerName="revoke-keys"
Oct 30 05:05:20.978292 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.978208    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa3c63a-eaf0-4aec-b463-adc66812e632" containerName="istio-proxy"
Oct 30 05:05:20.978292 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.978220    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ab9ae3-71d5-469d-8860-50240e90aa21" containerName="payload-pre-processing"
Oct 30 05:05:20.978292 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.978228    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d6aa4c-661b-4bb8-a051-87ed6f430464" containerName="payload-processing"
Oct 30 05:05:20.983254 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.983234    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:20.999610 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:20.999581    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"]
Oct 30 05:05:21.014819 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.007312    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:05:21.014819 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.007506    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:05:21.016538 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.016500    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1122785c-7n75f\" (UID: \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:21.016700 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.016680    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1122785c-7n75f\" (UID: \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:21.016802 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.016787    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9ns\" (UniqueName: \"kubernetes.io/projected/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-kube-api-access-9b9ns\") pod \"maas-api-revoke-sub-1122785c-7n75f\" (UID: \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:21.118455 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.118418    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1122785c-7n75f\" (UID: \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:21.118633 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.118559    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1122785c-7n75f\" (UID: \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:21.118690 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.118655    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9ns\" (UniqueName: \"kubernetes.io/projected/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-kube-api-access-9b9ns\") pod \"maas-api-revoke-sub-1122785c-7n75f\" (UID: \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:21.119142 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.119115    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1122785c-7n75f\" (UID: \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:21.121247 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.121225    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1122785c-7n75f\" (UID: \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:21.142504 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.142479    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9ns\" (UniqueName: \"kubernetes.io/projected/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-kube-api-access-9b9ns\") pod \"maas-api-revoke-sub-1122785c-7n75f\" (UID: \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:21.299783 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.299704    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:21.610459 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:05:21.610426    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8408b0f4_ac8e_4d4f_afaf_b67f547dcc41.slice/crio-806ed1f15e33aa5bb10b852478e2ef31d957bfc35b2aea2f91489f72890c4ea5 WatchSource:0}: Error finding container 806ed1f15e33aa5bb10b852478e2ef31d957bfc35b2aea2f91489f72890c4ea5: Status 404 returned error can't find the container with id 806ed1f15e33aa5bb10b852478e2ef31d957bfc35b2aea2f91489f72890c4ea5
Oct 30 05:05:21.612543 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:21.612517    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"]
Oct 30 05:05:22.417274 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:22.417191    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f" event={"ID":"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41","Type":"ContainerStarted","Data":"ff9508980ea7dc66fa78c4343fc472d5a382382cd015c3a6c3065d4f7bf94a90"}
Oct 30 05:05:22.417274 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:22.417243    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f" event={"ID":"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41","Type":"ContainerStarted","Data":"806ed1f15e33aa5bb10b852478e2ef31d957bfc35b2aea2f91489f72890c4ea5"}
Oct 30 05:05:22.460169 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:22.460118    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f" podStartSLOduration=2.460104049 podStartE2EDuration="2.460104049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:22.45752806 +0000 UTC m=+4102.198951158" watchObservedRunningTime="2026-10-30 05:05:22.460104049 +0000 UTC m=+4102.201527142"
Oct 30 05:05:23.427006 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:23.426973    5072 generic.go:356] "Generic (PLEG): container finished" podID="8408b0f4-ac8e-4d4f-afaf-b67f547dcc41" containerID="ff9508980ea7dc66fa78c4343fc472d5a382382cd015c3a6c3065d4f7bf94a90" exitCode=0
Oct 30 05:05:23.427448 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:23.427056    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f" event={"ID":"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41","Type":"ContainerDied","Data":"ff9508980ea7dc66fa78c4343fc472d5a382382cd015c3a6c3065d4f7bf94a90"}
Oct 30 05:05:24.595991 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:24.595963    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:24.683756 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:24.683707    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-kube-api-access-9b9ns\" (UniqueName: \"kubernetes.io/projected/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-kube-api-access-9b9ns\") pod \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\" (UID: \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\") "
Oct 30 05:05:24.683974 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:24.683892    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-openshift-ca\") pod \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\" (UID: \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\") "
Oct 30 05:05:24.683974 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:24.683921    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-xks-ca\") pod \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\" (UID: \"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41\") "
Oct 30 05:05:24.684334 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:24.684275    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-openshift-ca" pod "8408b0f4-ac8e-4d4f-afaf-b67f547dcc41" (UID: "8408b0f4-ac8e-4d4f-afaf-b67f547dcc41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:05:24.684537 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:24.684519    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:24.686156 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:24.686130    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-kube-api-access-9b9ns" pod "8408b0f4-ac8e-4d4f-afaf-b67f547dcc41" (UID: "8408b0f4-ac8e-4d4f-afaf-b67f547dcc41"). InnerVolumeSpecName "kube-api-access-9b9ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:05:24.686309 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:24.686287    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-xks-ca" pod "8408b0f4-ac8e-4d4f-afaf-b67f547dcc41" (UID: "8408b0f4-ac8e-4d4f-afaf-b67f547dcc41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:05:24.785953 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:24.785915    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:24.785953 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:24.785947    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b9ns\" (UniqueName: \"kubernetes.io/projected/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41-kube-api-access-9b9ns\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:25.443624 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:25.443591    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f" event={"ID":"8408b0f4-ac8e-4d4f-afaf-b67f547dcc41","Type":"ContainerDied","Data":"806ed1f15e33aa5bb10b852478e2ef31d957bfc35b2aea2f91489f72890c4ea5"}
Oct 30 05:05:25.443624 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:25.443601    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"
Oct 30 05:05:25.443852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:25.443650    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="806ed1f15e33aa5bb10b852478e2ef31d957bfc35b2aea2f91489f72890c4ea5"
Oct 30 05:05:26.998563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:26.998496    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-txlk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:05:26.999132 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:26.998600    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4" podUID="b748c4e2-2e95-43e6-8dc4-9bf3958a24d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:05:32.459992 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.459953    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"]
Oct 30 05:05:32.460763 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.460745    5072 cpu_manager.go:414] "containerMap: removing container" podUID="8408b0f4-ac8e-4d4f-afaf-b67f547dcc41" containerName="revoke-keys"
Oct 30 05:05:32.460763 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.460763    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8408b0f4-ac8e-4d4f-afaf-b67f547dcc41" containerName="revoke-keys"
Oct 30 05:05:32.460966 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.460951    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="8408b0f4-ac8e-4d4f-afaf-b67f547dcc41" containerName="revoke-keys"
Oct 30 05:05:32.465182 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.465162    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:32.485492 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.485462    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"]
Oct 30 05:05:32.491563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.491540    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:05:32.491745 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.491538    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:05:32.661442 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.661390    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67407bb-cbjp5\" (UID: \"114daf05-9ccf-4e38-82ad-f5859354f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:32.661965 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.661944    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67407bb-cbjp5\" (UID: \"114daf05-9ccf-4e38-82ad-f5859354f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:32.662132 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.662113    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjsh\" (UniqueName: \"kubernetes.io/projected/114daf05-9ccf-4e38-82ad-f5859354f86e-kube-api-access-jrjsh\") pod \"maas-api-revoke-sub-c67407bb-cbjp5\" (UID: \"114daf05-9ccf-4e38-82ad-f5859354f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:32.763822 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.763735    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67407bb-cbjp5\" (UID: \"114daf05-9ccf-4e38-82ad-f5859354f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:32.763979 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.763916    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67407bb-cbjp5\" (UID: \"114daf05-9ccf-4e38-82ad-f5859354f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:32.763979 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.763946    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjsh\" (UniqueName: \"kubernetes.io/projected/114daf05-9ccf-4e38-82ad-f5859354f86e-kube-api-access-jrjsh\") pod \"maas-api-revoke-sub-c67407bb-cbjp5\" (UID: \"114daf05-9ccf-4e38-82ad-f5859354f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:32.764621 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.764590    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67407bb-cbjp5\" (UID: \"114daf05-9ccf-4e38-82ad-f5859354f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:32.766340 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.766319    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67407bb-cbjp5\" (UID: \"114daf05-9ccf-4e38-82ad-f5859354f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:32.788830 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:32.788797    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjsh\" (UniqueName: \"kubernetes.io/projected/114daf05-9ccf-4e38-82ad-f5859354f86e-kube-api-access-jrjsh\") pod \"maas-api-revoke-sub-c67407bb-cbjp5\" (UID: \"114daf05-9ccf-4e38-82ad-f5859354f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:33.087256 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:33.087154    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:33.399440 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:05:33.399399    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114daf05_9ccf_4e38_82ad_f5859354f86e.slice/crio-89d677c08881df28566f6e2fe05dac199e891f552ef613aec4596b7e27b9244c WatchSource:0}: Error finding container 89d677c08881df28566f6e2fe05dac199e891f552ef613aec4596b7e27b9244c: Status 404 returned error can't find the container with id 89d677c08881df28566f6e2fe05dac199e891f552ef613aec4596b7e27b9244c
Oct 30 05:05:33.399604 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:33.399504    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"]
Oct 30 05:05:33.591419 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:33.591382    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5" event={"ID":"114daf05-9ccf-4e38-82ad-f5859354f86e","Type":"ContainerStarted","Data":"89d677c08881df28566f6e2fe05dac199e891f552ef613aec4596b7e27b9244c"}
Oct 30 05:05:34.602740 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:34.602709    5072 generic.go:356] "Generic (PLEG): container finished" podID="114daf05-9ccf-4e38-82ad-f5859354f86e" containerID="e03c6880e648676f8510235796ee66ac1ad579b02f831c13baf3af4bc0827ac1" exitCode=0
Oct 30 05:05:34.603182 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:34.602798    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5" event={"ID":"114daf05-9ccf-4e38-82ad-f5859354f86e","Type":"ContainerDied","Data":"e03c6880e648676f8510235796ee66ac1ad579b02f831c13baf3af4bc0827ac1"}
Oct 30 05:05:35.776130 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:35.776106    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:35.903417 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:35.903319    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-xks-ca\") pod \"114daf05-9ccf-4e38-82ad-f5859354f86e\" (UID: \"114daf05-9ccf-4e38-82ad-f5859354f86e\") "
Oct 30 05:05:35.903573 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:35.903543    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-openshift-ca\") pod \"114daf05-9ccf-4e38-82ad-f5859354f86e\" (UID: \"114daf05-9ccf-4e38-82ad-f5859354f86e\") "
Oct 30 05:05:35.903621 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:35.903597    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114daf05-9ccf-4e38-82ad-f5859354f86e-kube-api-access-jrjsh\" (UniqueName: \"kubernetes.io/projected/114daf05-9ccf-4e38-82ad-f5859354f86e-kube-api-access-jrjsh\") pod \"114daf05-9ccf-4e38-82ad-f5859354f86e\" (UID: \"114daf05-9ccf-4e38-82ad-f5859354f86e\") "
Oct 30 05:05:35.903909 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:35.903880    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-openshift-ca" pod "114daf05-9ccf-4e38-82ad-f5859354f86e" (UID: "114daf05-9ccf-4e38-82ad-f5859354f86e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:05:35.904234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:35.904043    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:35.905784 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:35.905754    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-xks-ca" pod "114daf05-9ccf-4e38-82ad-f5859354f86e" (UID: "114daf05-9ccf-4e38-82ad-f5859354f86e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:05:35.905909 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:35.905780    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114daf05-9ccf-4e38-82ad-f5859354f86e-kube-api-access-jrjsh" pod "114daf05-9ccf-4e38-82ad-f5859354f86e" (UID: "114daf05-9ccf-4e38-82ad-f5859354f86e"). InnerVolumeSpecName "kube-api-access-jrjsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:05:36.005484 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:36.005443    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrjsh\" (UniqueName: \"kubernetes.io/projected/114daf05-9ccf-4e38-82ad-f5859354f86e-kube-api-access-jrjsh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:36.005484 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:36.005484    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114daf05-9ccf-4e38-82ad-f5859354f86e-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:36.627637 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:36.627610    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"
Oct 30 05:05:36.628138 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:36.627599    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5" event={"ID":"114daf05-9ccf-4e38-82ad-f5859354f86e","Type":"ContainerDied","Data":"89d677c08881df28566f6e2fe05dac199e891f552ef613aec4596b7e27b9244c"}
Oct 30 05:05:36.628220 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:36.628177    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89d677c08881df28566f6e2fe05dac199e891f552ef613aec4596b7e27b9244c"
Oct 30 05:05:41.753021 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:05:41.752978    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:05:42.100800 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.100718    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"]
Oct 30 05:05:42.101944 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.101914    5072 cpu_manager.go:414] "containerMap: removing container" podUID="114daf05-9ccf-4e38-82ad-f5859354f86e" containerName="revoke-keys"
Oct 30 05:05:42.101944 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.101942    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="114daf05-9ccf-4e38-82ad-f5859354f86e" containerName="revoke-keys"
Oct 30 05:05:42.102228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.102206    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="114daf05-9ccf-4e38-82ad-f5859354f86e" containerName="revoke-keys"
Oct 30 05:05:42.106468 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.106448    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.129269 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.129239    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"]
Oct 30 05:05:42.189361 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.189326    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.189514 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.189448    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.189514 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.189495    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5a918c-313e-446c-be37-2f71e771ca98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.189615 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.189533    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.189658 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.189646    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvkzk\" (UniqueName: \"kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-kube-api-access-kvkzk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.189696 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.189678    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.189732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.189704    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.189799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.189778    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a918c-313e-446c-be37-2f71e771ca98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.189836 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.189822    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.291518 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.291481    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.291703 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.291544    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvkzk\" (UniqueName: \"kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-kube-api-access-kvkzk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.291703 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.291577    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.291703 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.291611    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.291703 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.291635    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a918c-313e-446c-be37-2f71e771ca98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.291703 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.291665    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.292025 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.291803    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.292025 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.291896    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.292025 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.291942    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5a918c-313e-446c-be37-2f71e771ca98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.292025 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.291953    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.292527 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.292032    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.292527 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.292168    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.292527 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.292259    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.292740 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.292551    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a918c-313e-446c-be37-2f71e771ca98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.294288 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.294263    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.294703 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.294687    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5a918c-313e-446c-be37-2f71e771ca98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.315603 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.315576    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.315709 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.315638    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvkzk\" (UniqueName: \"kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-kube-api-access-kvkzk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pghfc\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.426004 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.425928    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:42.751403 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:05:42.751369    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5a918c_313e_446c_be37_2f71e771ca98.slice/crio-f7243fc77a156562ab277bcfa208edfbd8afb52ac9522c7c6bd3d7cd7fe902e0 WatchSource:0}: Error finding container f7243fc77a156562ab277bcfa208edfbd8afb52ac9522c7c6bd3d7cd7fe902e0: Status 404 returned error can't find the container with id f7243fc77a156562ab277bcfa208edfbd8afb52ac9522c7c6bd3d7cd7fe902e0
Oct 30 05:05:42.751614 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.751591    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"]
Oct 30 05:05:42.755914 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.755860    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:05:42.756330 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.756024    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:05:42.767898 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:42.767852    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:05:43.691716 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:43.691676    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" event={"ID":"6a5a918c-313e-446c-be37-2f71e771ca98","Type":"ContainerStarted","Data":"03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f"}
Oct 30 05:05:43.691939 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:43.691747    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" event={"ID":"6a5a918c-313e-446c-be37-2f71e771ca98","Type":"ContainerStarted","Data":"f7243fc77a156562ab277bcfa208edfbd8afb52ac9522c7c6bd3d7cd7fe902e0"}
Oct 30 05:05:43.780178 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:43.776783    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" podStartSLOduration=1.77676221 podStartE2EDuration="1.77676221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:43.749877639 +0000 UTC m=+4123.491300748" watchObservedRunningTime="2026-10-30 05:05:43.77676221 +0000 UTC m=+4123.518185308"
Oct 30 05:05:44.426309 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:44.426276    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:44.428364 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:44.428335    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pghfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 05:05:44.428510 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:44.428386    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" podUID="6a5a918c-313e-446c-be37-2f71e771ca98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 05:05:45.427234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:45.427196    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pghfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 05:05:45.427835 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:45.427250    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" podUID="6a5a918c-313e-446c-be37-2f71e771ca98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 05:05:46.426882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:46.426844    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pghfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 05:05:46.427062 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:46.426905    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" podUID="6a5a918c-313e-446c-be37-2f71e771ca98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 05:05:47.426748 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:47.426714    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pghfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 05:05:47.427183 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:47.426765    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" podUID="6a5a918c-313e-446c-be37-2f71e771ca98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 05:05:48.226184 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.226148    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"]
Oct 30 05:05:48.231125 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.231096    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:48.250338 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.250305    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"]
Oct 30 05:05:48.264222 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.263728    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:05:48.264222 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.263773    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:05:48.404676 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.404642    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b4b686-z5btt\" (UID: \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:48.404846 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.404778    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4kqz\" (UniqueName: \"kubernetes.io/projected/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-kube-api-access-z4kqz\") pod \"maas-api-revoke-sub-98b4b686-z5btt\" (UID: \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:48.405118 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.405057    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b4b686-z5btt\" (UID: \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:48.426717 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.426686    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pghfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 05:05:48.426866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.426739    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" podUID="6a5a918c-313e-446c-be37-2f71e771ca98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 05:05:48.507098 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.506988    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b4b686-z5btt\" (UID: \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:48.507283 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.507151    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kqz\" (UniqueName: \"kubernetes.io/projected/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-kube-api-access-z4kqz\") pod \"maas-api-revoke-sub-98b4b686-z5btt\" (UID: \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:48.507378 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.507354    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b4b686-z5btt\" (UID: \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:48.508040 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.508010    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b4b686-z5btt\" (UID: \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:48.509705 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.509684    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b4b686-z5btt\" (UID: \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:48.531189 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.531167    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4kqz\" (UniqueName: \"kubernetes.io/projected/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-kube-api-access-z4kqz\") pod \"maas-api-revoke-sub-98b4b686-z5btt\" (UID: \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:48.554811 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.554785    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:48.896188 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:48.896156    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"]
Oct 30 05:05:48.896342 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:05:48.896300    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d426b5b_3875_48b9_8c8c_cc9deb7e68a7.slice/crio-3ce31dc59fe305dea0f3564936df1d903bf648b8fd14cb07af31589d1fa71857 WatchSource:0}: Error finding container 3ce31dc59fe305dea0f3564936df1d903bf648b8fd14cb07af31589d1fa71857: Status 404 returned error can't find the container with id 3ce31dc59fe305dea0f3564936df1d903bf648b8fd14cb07af31589d1fa71857
Oct 30 05:05:49.014294 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.014255    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"]
Oct 30 05:05:49.209183 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.207007    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"]
Oct 30 05:05:49.257102 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.248304    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"]
Oct 30 05:05:49.257102 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.248417    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:49.330913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.330869    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b49af0c-h69r4\" (UID: \"d205d118-eeda-400a-9338-a28f9aac7a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:49.331141 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.331041    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b49af0c-h69r4\" (UID: \"d205d118-eeda-400a-9338-a28f9aac7a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:49.331209 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.331179    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rrz\" (UniqueName: \"kubernetes.io/projected/d205d118-eeda-400a-9338-a28f9aac7a61-kube-api-access-w2rrz\") pod \"maas-api-revoke-sub-5b49af0c-h69r4\" (UID: \"d205d118-eeda-400a-9338-a28f9aac7a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:49.427760 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.427716    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pghfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 05:05:49.428260 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.427782    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" podUID="6a5a918c-313e-446c-be37-2f71e771ca98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 05:05:49.432745 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.432711    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b49af0c-h69r4\" (UID: \"d205d118-eeda-400a-9338-a28f9aac7a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:49.432907 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.432873    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rrz\" (UniqueName: \"kubernetes.io/projected/d205d118-eeda-400a-9338-a28f9aac7a61-kube-api-access-w2rrz\") pod \"maas-api-revoke-sub-5b49af0c-h69r4\" (UID: \"d205d118-eeda-400a-9338-a28f9aac7a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:49.433041 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.433018    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b49af0c-h69r4\" (UID: \"d205d118-eeda-400a-9338-a28f9aac7a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:49.433624 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.433601    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b49af0c-h69r4\" (UID: \"d205d118-eeda-400a-9338-a28f9aac7a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:49.435561 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.435546    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b49af0c-h69r4\" (UID: \"d205d118-eeda-400a-9338-a28f9aac7a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:49.457402 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.457323    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rrz\" (UniqueName: \"kubernetes.io/projected/d205d118-eeda-400a-9338-a28f9aac7a61-kube-api-access-w2rrz\") pod \"maas-api-revoke-sub-5b49af0c-h69r4\" (UID: \"d205d118-eeda-400a-9338-a28f9aac7a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:49.575973 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.575941    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:49.763031 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.762299    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt" event={"ID":"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7","Type":"ContainerStarted","Data":"1fcff1b5e15fc389ab7b97252cdd9d223c5bc33ca4003f37d9ab4be3398e6133"}
Oct 30 05:05:49.763031 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.762364    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt" event={"ID":"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7","Type":"ContainerStarted","Data":"3ce31dc59fe305dea0f3564936df1d903bf648b8fd14cb07af31589d1fa71857"}
Oct 30 05:05:49.806262 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.806209    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt" podStartSLOduration=1.806195212 podStartE2EDuration="1.806195212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:49.803489512 +0000 UTC m=+4129.544912611" watchObservedRunningTime="2026-10-30 05:05:49.806195212 +0000 UTC m=+4129.547618307"
Oct 30 05:05:49.909965 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:05:49.909931    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd205d118_eeda_400a_9338_a28f9aac7a61.slice/crio-c95cbb286162ede7435b4496804e23959b20fa08c2beec4b69c2cab97507cad1 WatchSource:0}: Error finding container c95cbb286162ede7435b4496804e23959b20fa08c2beec4b69c2cab97507cad1: Status 404 returned error can't find the container with id c95cbb286162ede7435b4496804e23959b20fa08c2beec4b69c2cab97507cad1
Oct 30 05:05:49.910363 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:49.910336    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"]
Oct 30 05:05:50.427132 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:50.427048    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pghfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 05:05:50.427132 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:50.427117    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" podUID="6a5a918c-313e-446c-be37-2f71e771ca98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 05:05:50.773118 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:50.773052    5072 generic.go:356] "Generic (PLEG): container finished" podID="8d426b5b-3875-48b9-8c8c-cc9deb7e68a7" containerID="1fcff1b5e15fc389ab7b97252cdd9d223c5bc33ca4003f37d9ab4be3398e6133" exitCode=0
Oct 30 05:05:50.773606 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:50.773116    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt" event={"ID":"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7","Type":"ContainerDied","Data":"1fcff1b5e15fc389ab7b97252cdd9d223c5bc33ca4003f37d9ab4be3398e6133"}
Oct 30 05:05:50.775177 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:50.775150    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4" event={"ID":"d205d118-eeda-400a-9338-a28f9aac7a61","Type":"ContainerStarted","Data":"d71669cfaf70b96d50bc6953fffce1569de269078e07eb8b84f0bc627d12a7c0"}
Oct 30 05:05:50.775317 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:50.775212    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4" event={"ID":"d205d118-eeda-400a-9338-a28f9aac7a61","Type":"ContainerStarted","Data":"c95cbb286162ede7435b4496804e23959b20fa08c2beec4b69c2cab97507cad1"}
Oct 30 05:05:50.876696 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:50.876136    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4" podStartSLOduration=1.87611101 podStartE2EDuration="1.87611101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:50.85979566 +0000 UTC m=+4130.601218758" watchObservedRunningTime="2026-10-30 05:05:50.87611101 +0000 UTC m=+4130.617534108"
Oct 30 05:05:51.559020 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.558986    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:51.559562 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.559524    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:51.560548 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.560524    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:05:51.792196 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.792157    5072 generic.go:356] "Generic (PLEG): container finished" podID="d205d118-eeda-400a-9338-a28f9aac7a61" containerID="d71669cfaf70b96d50bc6953fffce1569de269078e07eb8b84f0bc627d12a7c0" exitCode=0
Oct 30 05:05:51.792670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.792257    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4" event={"ID":"d205d118-eeda-400a-9338-a28f9aac7a61","Type":"ContainerDied","Data":"d71669cfaf70b96d50bc6953fffce1569de269078e07eb8b84f0bc627d12a7c0"}
Oct 30 05:05:51.983835 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.983805    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:51.996033 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.996006    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-openshift-ca\") pod \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\" (UID: \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\") "
Oct 30 05:05:51.996193 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.996150    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-kube-api-access-z4kqz\" (UniqueName: \"kubernetes.io/projected/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-kube-api-access-z4kqz\") pod \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\" (UID: \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\") "
Oct 30 05:05:51.996239 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.996213    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-xks-ca\") pod \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\" (UID: \"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7\") "
Oct 30 05:05:51.996449 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.996410    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-openshift-ca" pod "8d426b5b-3875-48b9-8c8c-cc9deb7e68a7" (UID: "8d426b5b-3875-48b9-8c8c-cc9deb7e68a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:05:51.996718 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.996694    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:51.998440 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.998418    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-kube-api-access-z4kqz" pod "8d426b5b-3875-48b9-8c8c-cc9deb7e68a7" (UID: "8d426b5b-3875-48b9-8c8c-cc9deb7e68a7"). InnerVolumeSpecName "kube-api-access-z4kqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:05:51.998535 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:51.998514    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-xks-ca" pod "8d426b5b-3875-48b9-8c8c-cc9deb7e68a7" (UID: "8d426b5b-3875-48b9-8c8c-cc9deb7e68a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:05:52.097640 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:52.097607    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4kqz\" (UniqueName: \"kubernetes.io/projected/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-kube-api-access-z4kqz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:52.097640 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:52.097641    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:52.801180 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:52.801130    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt" event={"ID":"8d426b5b-3875-48b9-8c8c-cc9deb7e68a7","Type":"ContainerDied","Data":"3ce31dc59fe305dea0f3564936df1d903bf648b8fd14cb07af31589d1fa71857"}
Oct 30 05:05:52.801592 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:52.801198    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ce31dc59fe305dea0f3564936df1d903bf648b8fd14cb07af31589d1fa71857"
Oct 30 05:05:52.801592 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:52.801155    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"
Oct 30 05:05:52.961979 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:52.961954    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:53.007195 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:53.007165    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-openshift-ca\") pod \"d205d118-eeda-400a-9338-a28f9aac7a61\" (UID: \"d205d118-eeda-400a-9338-a28f9aac7a61\") "
Oct 30 05:05:53.007626 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:53.007462    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-xks-ca\") pod \"d205d118-eeda-400a-9338-a28f9aac7a61\" (UID: \"d205d118-eeda-400a-9338-a28f9aac7a61\") "
Oct 30 05:05:53.007626 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:53.007516    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d205d118-eeda-400a-9338-a28f9aac7a61-kube-api-access-w2rrz\" (UniqueName: \"kubernetes.io/projected/d205d118-eeda-400a-9338-a28f9aac7a61-kube-api-access-w2rrz\") pod \"d205d118-eeda-400a-9338-a28f9aac7a61\" (UID: \"d205d118-eeda-400a-9338-a28f9aac7a61\") "
Oct 30 05:05:53.008381 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:53.008353    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-openshift-ca" pod "d205d118-eeda-400a-9338-a28f9aac7a61" (UID: "d205d118-eeda-400a-9338-a28f9aac7a61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:05:53.012127 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:53.010954    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-xks-ca" pod "d205d118-eeda-400a-9338-a28f9aac7a61" (UID: "d205d118-eeda-400a-9338-a28f9aac7a61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:05:53.012127 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:53.010961    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d205d118-eeda-400a-9338-a28f9aac7a61-kube-api-access-w2rrz" pod "d205d118-eeda-400a-9338-a28f9aac7a61" (UID: "d205d118-eeda-400a-9338-a28f9aac7a61"). InnerVolumeSpecName "kube-api-access-w2rrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:05:53.109047 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:53.108980    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:53.109047 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:53.109013    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rrz\" (UniqueName: \"kubernetes.io/projected/d205d118-eeda-400a-9338-a28f9aac7a61-kube-api-access-w2rrz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:53.109047 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:53.109024    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d205d118-eeda-400a-9338-a28f9aac7a61-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:05:53.810569 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:53.810541    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"
Oct 30 05:05:53.810985 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:53.810574    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4" event={"ID":"d205d118-eeda-400a-9338-a28f9aac7a61","Type":"ContainerDied","Data":"c95cbb286162ede7435b4496804e23959b20fa08c2beec4b69c2cab97507cad1"}
Oct 30 05:05:53.810985 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:53.810622    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c95cbb286162ede7435b4496804e23959b20fa08c2beec4b69c2cab97507cad1"
Oct 30 05:05:54.923674 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:54.923639    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"]
Oct 30 05:05:54.957352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:54.957320    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"]
Oct 30 05:05:59.924534 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:59.924490    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45" podUID="7b81fac8-afdb-4c6b-a95a-4654a89d9b31" containerName="payload-processing" containerID="cri-o://2399637d24a925a3a9b640708d700cca0f44a7923f6bfbec0f9bb21fbdc42334" gracePeriod=25
Oct 30 05:05:59.958189 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:05:59.958146    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf" podUID="aa34040e-b825-4490-bad5-637800da41e2" containerName="payload-pre-processing" containerID="cri-o://f735aa6bf10267b66dc3fc63b04555b0935e28f53936a4c6eccd0b9ff5eafe29" gracePeriod=25
Oct 30 05:06:00.955118 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:00.955052    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45" podUID="7b81fac8-afdb-4c6b-a95a-4654a89d9b31" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:06:00.955514 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:00.955051    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf" podUID="aa34040e-b825-4490-bad5-637800da41e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:06:01.030634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:01.030598    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"]
Oct 30 05:06:01.042903 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:01.042870    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7dc8b74-26jtf"]
Oct 30 05:06:02.032613 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:02.032574    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"]
Oct 30 05:06:02.045381 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:02.045353    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d542375d-nhd2n"]
Oct 30 05:06:02.410309 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:02.410232    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c7ea18-975b-4f96-80aa-a4f20358d963" path="/var/lib/kubelet/pods/a5c7ea18-975b-4f96-80aa-a4f20358d963/volumes"
Oct 30 05:06:02.410735 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:02.410721    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e595a127-8671-4ba9-a20f-7fbe5d92f3c6" path="/var/lib/kubelet/pods/e595a127-8671-4ba9-a20f-7fbe5d92f3c6/volumes"
Oct 30 05:06:04.080689 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.080651    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h" podUID="0c4a4c41-7fff-4875-9de2-ee671ed2a976" containerName="main" containerID="cri-o://ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153" gracePeriod=45
Oct 30 05:06:04.394427 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.394398    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:06:04.484201 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.484162    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tls-certs\") pod \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") "
Oct 30 05:06:04.484391 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.484246    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-model-cache\") pod \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") "
Oct 30 05:06:04.484391 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.484344    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4a4c41-7fff-4875-9de2-ee671ed2a976-kube-api-access-wxz48\" (UniqueName: \"kubernetes.io/projected/0c4a4c41-7fff-4875-9de2-ee671ed2a976-kube-api-access-wxz48\") pod \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") "
Oct 30 05:06:04.484391 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.484379    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-dshm\") pod \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") "
Oct 30 05:06:04.484604 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.484485    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tmp-dir\") pod \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") "
Oct 30 05:06:04.484604 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.484553    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-model-cache" pod "0c4a4c41-7fff-4875-9de2-ee671ed2a976" (UID: "0c4a4c41-7fff-4875-9de2-ee671ed2a976"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:04.484604 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.484598    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-home\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-home\") pod \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\" (UID: \"0c4a4c41-7fff-4875-9de2-ee671ed2a976\") "
Oct 30 05:06:04.484875 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.484850    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tmp-dir" pod "0c4a4c41-7fff-4875-9de2-ee671ed2a976" (UID: "0c4a4c41-7fff-4875-9de2-ee671ed2a976"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:04.485048 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.485025    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-home" pod "0c4a4c41-7fff-4875-9de2-ee671ed2a976" (UID: "0c4a4c41-7fff-4875-9de2-ee671ed2a976"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:04.485278 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.485260    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:04.485348 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.485283    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:04.485348 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.485296    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:04.487179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.487148    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4a4c41-7fff-4875-9de2-ee671ed2a976-kube-api-access-wxz48" pod "0c4a4c41-7fff-4875-9de2-ee671ed2a976" (UID: "0c4a4c41-7fff-4875-9de2-ee671ed2a976"). InnerVolumeSpecName "kube-api-access-wxz48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:06:04.487492 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.487453    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-dshm" pod "0c4a4c41-7fff-4875-9de2-ee671ed2a976" (UID: "0c4a4c41-7fff-4875-9de2-ee671ed2a976"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:04.488480 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.488453    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tls-certs" pod "0c4a4c41-7fff-4875-9de2-ee671ed2a976" (UID: "0c4a4c41-7fff-4875-9de2-ee671ed2a976"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:06:04.587022 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.586982    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4a4c41-7fff-4875-9de2-ee671ed2a976-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:04.587022 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.587027    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxz48\" (UniqueName: \"kubernetes.io/projected/0c4a4c41-7fff-4875-9de2-ee671ed2a976-kube-api-access-wxz48\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:04.587022 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.587041    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4a4c41-7fff-4875-9de2-ee671ed2a976-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:04.933124 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.933048    5072 generic.go:356] "Generic (PLEG): container finished" podID="0c4a4c41-7fff-4875-9de2-ee671ed2a976" containerID="ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153" exitCode=0
Oct 30 05:06:04.933340 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.933213    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"
Oct 30 05:06:04.933340 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.933246    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h" event={"ID":"0c4a4c41-7fff-4875-9de2-ee671ed2a976","Type":"ContainerDied","Data":"ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153"}
Oct 30 05:06:04.933340 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.933300    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h" event={"ID":"0c4a4c41-7fff-4875-9de2-ee671ed2a976","Type":"ContainerDied","Data":"b08ad1bd37656a96d6f3bc296fb873de71fe5d2a6ad36d85149826f7ec1655e5"}
Oct 30 05:06:04.933340 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.933320    5072 scope.go:122] "RemoveContainer" containerID="ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153"
Oct 30 05:06:04.984107 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.983936    5072 scope.go:122] "RemoveContainer" containerID="ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153"
Oct 30 05:06:04.987112 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:06:04.984425    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153\": container with ID starting with ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153 not found: ID does not exist" containerID="ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153"
Oct 30 05:06:04.987112 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:04.984479    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153"} err="failed to get container status \"ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153\": rpc error: code = NotFound desc = could not find container \"ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153\": container with ID starting with ff42bb0a52294cd900156125a5d57e51c4834c5171840113fe175862c216f153 not found: ID does not exist"
Oct 30 05:06:05.043102 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.042127    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"]
Oct 30 05:06:05.060176 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.059651    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e592d7/test-model-e592d7-kserve-7c8d5597d7-sns2h"]
Oct 30 05:06:05.396998 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.396962    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"]
Oct 30 05:06:05.397827 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.397806    5072 cpu_manager.go:414] "containerMap: removing container" podUID="0c4a4c41-7fff-4875-9de2-ee671ed2a976" containerName="main"
Oct 30 05:06:05.397908 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.397829    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4a4c41-7fff-4875-9de2-ee671ed2a976" containerName="main"
Oct 30 05:06:05.397908 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.397867    5072 cpu_manager.go:414] "containerMap: removing container" podUID="8d426b5b-3875-48b9-8c8c-cc9deb7e68a7" containerName="revoke-keys"
Oct 30 05:06:05.397908 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.397872    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d426b5b-3875-48b9-8c8c-cc9deb7e68a7" containerName="revoke-keys"
Oct 30 05:06:05.397908 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.397881    5072 cpu_manager.go:414] "containerMap: removing container" podUID="d205d118-eeda-400a-9338-a28f9aac7a61" containerName="revoke-keys"
Oct 30 05:06:05.397908 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.397888    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d205d118-eeda-400a-9338-a28f9aac7a61" containerName="revoke-keys"
Oct 30 05:06:05.398207 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.398013    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d426b5b-3875-48b9-8c8c-cc9deb7e68a7" containerName="revoke-keys"
Oct 30 05:06:05.398207 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.398022    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d205d118-eeda-400a-9338-a28f9aac7a61" containerName="revoke-keys"
Oct 30 05:06:05.398207 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.398039    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4a4c41-7fff-4875-9de2-ee671ed2a976" containerName="main"
Oct 30 05:06:05.404234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.404206    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:05.420231 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.420203    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"]
Oct 30 05:06:05.425217 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.425065    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:06:05.425217 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.425182    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:06:05.507217 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.507184    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4dpd\" (UniqueName: \"kubernetes.io/projected/5806f630-df78-471c-a399-e3c6413f370b-kube-api-access-b4dpd\") pod \"maas-api-revoke-sub-ac28b54c-2cntk\" (UID: \"5806f630-df78-471c-a399-e3c6413f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:05.507434 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.507228    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5806f630-df78-471c-a399-e3c6413f370b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac28b54c-2cntk\" (UID: \"5806f630-df78-471c-a399-e3c6413f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:05.507434 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.507328    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5806f630-df78-471c-a399-e3c6413f370b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac28b54c-2cntk\" (UID: \"5806f630-df78-471c-a399-e3c6413f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:05.609215 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.609184    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4dpd\" (UniqueName: \"kubernetes.io/projected/5806f630-df78-471c-a399-e3c6413f370b-kube-api-access-b4dpd\") pod \"maas-api-revoke-sub-ac28b54c-2cntk\" (UID: \"5806f630-df78-471c-a399-e3c6413f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:05.609410 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.609220    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5806f630-df78-471c-a399-e3c6413f370b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac28b54c-2cntk\" (UID: \"5806f630-df78-471c-a399-e3c6413f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:05.609410 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.609254    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5806f630-df78-471c-a399-e3c6413f370b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac28b54c-2cntk\" (UID: \"5806f630-df78-471c-a399-e3c6413f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:05.609989 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.609969    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5806f630-df78-471c-a399-e3c6413f370b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac28b54c-2cntk\" (UID: \"5806f630-df78-471c-a399-e3c6413f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:05.611938 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.611912    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5806f630-df78-471c-a399-e3c6413f370b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac28b54c-2cntk\" (UID: \"5806f630-df78-471c-a399-e3c6413f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:05.633563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.633535    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4dpd\" (UniqueName: \"kubernetes.io/projected/5806f630-df78-471c-a399-e3c6413f370b-kube-api-access-b4dpd\") pod \"maas-api-revoke-sub-ac28b54c-2cntk\" (UID: \"5806f630-df78-471c-a399-e3c6413f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:05.725744 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:05.725708    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:06.034545 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:06:06.034516    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5806f630_df78_471c_a399_e3c6413f370b.slice/crio-6875b93854111cfcedf3b212776382087414a226863cd2c27404cfb071b7455f WatchSource:0}: Error finding container 6875b93854111cfcedf3b212776382087414a226863cd2c27404cfb071b7455f: Status 404 returned error can't find the container with id 6875b93854111cfcedf3b212776382087414a226863cd2c27404cfb071b7455f
Oct 30 05:06:06.039631 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:06.035387    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"]
Oct 30 05:06:06.412560 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:06.412526    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4a4c41-7fff-4875-9de2-ee671ed2a976" path="/var/lib/kubelet/pods/0c4a4c41-7fff-4875-9de2-ee671ed2a976/volumes"
Oct 30 05:06:06.954426 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:06.954383    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk" event={"ID":"5806f630-df78-471c-a399-e3c6413f370b","Type":"ContainerStarted","Data":"d259ff17b82a7512891f3becb13fe685f8dd6be3c911c5636e518c2b3d13c15c"}
Oct 30 05:06:06.954756 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:06.954724    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk" event={"ID":"5806f630-df78-471c-a399-e3c6413f370b","Type":"ContainerStarted","Data":"6875b93854111cfcedf3b212776382087414a226863cd2c27404cfb071b7455f"}
Oct 30 05:06:06.998322 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:06.998266    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk" podStartSLOduration=1.998252442 podStartE2EDuration="1.998252442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:06.996921182 +0000 UTC m=+4146.738344281" watchObservedRunningTime="2026-10-30 05:06:06.998252442 +0000 UTC m=+4146.739675538"
Oct 30 05:06:07.964291 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:07.964261    5072 generic.go:356] "Generic (PLEG): container finished" podID="5806f630-df78-471c-a399-e3c6413f370b" containerID="d259ff17b82a7512891f3becb13fe685f8dd6be3c911c5636e518c2b3d13c15c" exitCode=0
Oct 30 05:06:07.964710 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:07.964387    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk" event={"ID":"5806f630-df78-471c-a399-e3c6413f370b","Type":"ContainerDied","Data":"d259ff17b82a7512891f3becb13fe685f8dd6be3c911c5636e518c2b3d13c15c"}
Oct 30 05:06:09.132671 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.132645    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:09.260735 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.260655    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5806f630-df78-471c-a399-e3c6413f370b-kube-api-access-b4dpd\" (UniqueName: \"kubernetes.io/projected/5806f630-df78-471c-a399-e3c6413f370b-kube-api-access-b4dpd\") pod \"5806f630-df78-471c-a399-e3c6413f370b\" (UID: \"5806f630-df78-471c-a399-e3c6413f370b\") "
Oct 30 05:06:09.260879 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.260750    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5806f630-df78-471c-a399-e3c6413f370b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5806f630-df78-471c-a399-e3c6413f370b-maas-api-openshift-ca\") pod \"5806f630-df78-471c-a399-e3c6413f370b\" (UID: \"5806f630-df78-471c-a399-e3c6413f370b\") "
Oct 30 05:06:09.260879 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.260802    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5806f630-df78-471c-a399-e3c6413f370b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5806f630-df78-471c-a399-e3c6413f370b-maas-api-xks-ca\") pod \"5806f630-df78-471c-a399-e3c6413f370b\" (UID: \"5806f630-df78-471c-a399-e3c6413f370b\") "
Oct 30 05:06:09.261205 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.261156    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5806f630-df78-471c-a399-e3c6413f370b-maas-api-openshift-ca" pod "5806f630-df78-471c-a399-e3c6413f370b" (UID: "5806f630-df78-471c-a399-e3c6413f370b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:06:09.261554 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.261482    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5806f630-df78-471c-a399-e3c6413f370b-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:09.263066 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.263040    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5806f630-df78-471c-a399-e3c6413f370b-kube-api-access-b4dpd" pod "5806f630-df78-471c-a399-e3c6413f370b" (UID: "5806f630-df78-471c-a399-e3c6413f370b"). InnerVolumeSpecName "kube-api-access-b4dpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:06:09.263191 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.263102    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5806f630-df78-471c-a399-e3c6413f370b-maas-api-xks-ca" pod "5806f630-df78-471c-a399-e3c6413f370b" (UID: "5806f630-df78-471c-a399-e3c6413f370b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:06:09.362554 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.362517    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4dpd\" (UniqueName: \"kubernetes.io/projected/5806f630-df78-471c-a399-e3c6413f370b-kube-api-access-b4dpd\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:09.362554 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.362555    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5806f630-df78-471c-a399-e3c6413f370b-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:09.979997 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.979968    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"
Oct 30 05:06:09.980205 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.979960    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk" event={"ID":"5806f630-df78-471c-a399-e3c6413f370b","Type":"ContainerDied","Data":"6875b93854111cfcedf3b212776382087414a226863cd2c27404cfb071b7455f"}
Oct 30 05:06:09.980205 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:09.980111    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6875b93854111cfcedf3b212776382087414a226863cd2c27404cfb071b7455f"
Oct 30 05:06:10.954768 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:10.954732    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf" podUID="aa34040e-b825-4490-bad5-637800da41e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:06:10.954768 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:10.954756    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45" podUID="7b81fac8-afdb-4c6b-a95a-4654a89d9b31" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:06:11.753289 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:06:11.753241    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:06:13.030428 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:13.030388    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"]
Oct 30 05:06:13.067653 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:13.067617    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a231777e-t86nv"]
Oct 30 05:06:14.410131 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:14.410101    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9890a3c3-f015-4c5a-a74d-2d241cb9742b" path="/var/lib/kubelet/pods/9890a3c3-f015-4c5a-a74d-2d241cb9742b/volumes"
Oct 30 05:06:15.040007 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:15.039975    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"]
Oct 30 05:06:15.052731 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:15.052701    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69bdb975-vtbzp"]
Oct 30 05:06:16.410632 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:16.410601    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c413afde-5c6c-43b2-af90-49ad91009908" path="/var/lib/kubelet/pods/c413afde-5c6c-43b2-af90-49ad91009908/volumes"
Oct 30 05:06:20.116913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.116703    5072 generic.go:356] "Generic (PLEG): container finished" podID="7b81fac8-afdb-4c6b-a95a-4654a89d9b31" containerID="2399637d24a925a3a9b640708d700cca0f44a7923f6bfbec0f9bb21fbdc42334" exitCode=0
Oct 30 05:06:20.116913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.116833    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45" event={"ID":"7b81fac8-afdb-4c6b-a95a-4654a89d9b31","Type":"ContainerDied","Data":"2399637d24a925a3a9b640708d700cca0f44a7923f6bfbec0f9bb21fbdc42334"}
Oct 30 05:06:20.120352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.120196    5072 generic.go:356] "Generic (PLEG): container finished" podID="aa34040e-b825-4490-bad5-637800da41e2" containerID="f735aa6bf10267b66dc3fc63b04555b0935e28f53936a4c6eccd0b9ff5eafe29" exitCode=0
Oct 30 05:06:20.120352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.120248    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf" event={"ID":"aa34040e-b825-4490-bad5-637800da41e2","Type":"ContainerDied","Data":"f735aa6bf10267b66dc3fc63b04555b0935e28f53936a4c6eccd0b9ff5eafe29"}
Oct 30 05:06:20.352981 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.352953    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"
Oct 30 05:06:20.376688 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.376665    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"
Oct 30 05:06:20.450347 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.450313    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-plugins-config-volume\") pod \"7b81fac8-afdb-4c6b-a95a-4654a89d9b31\" (UID: \"7b81fac8-afdb-4c6b-a95a-4654a89d9b31\") "
Oct 30 05:06:20.450530 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.450370    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa34040e-b825-4490-bad5-637800da41e2-kube-api-access-2px6w\" (UniqueName: \"kubernetes.io/projected/aa34040e-b825-4490-bad5-637800da41e2-kube-api-access-2px6w\") pod \"aa34040e-b825-4490-bad5-637800da41e2\" (UID: \"aa34040e-b825-4490-bad5-637800da41e2\") "
Oct 30 05:06:20.450530 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.450411    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa34040e-b825-4490-bad5-637800da41e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa34040e-b825-4490-bad5-637800da41e2-plugins-config-volume\") pod \"aa34040e-b825-4490-bad5-637800da41e2\" (UID: \"aa34040e-b825-4490-bad5-637800da41e2\") "
Oct 30 05:06:20.450530 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.450473    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-kube-api-access-6xx9v\" (UniqueName: \"kubernetes.io/projected/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-kube-api-access-6xx9v\") pod \"7b81fac8-afdb-4c6b-a95a-4654a89d9b31\" (UID: \"7b81fac8-afdb-4c6b-a95a-4654a89d9b31\") "
Oct 30 05:06:20.451500 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.451472    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-plugins-config-volume" pod "7b81fac8-afdb-4c6b-a95a-4654a89d9b31" (UID: "7b81fac8-afdb-4c6b-a95a-4654a89d9b31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:06:20.452626 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.452597    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa34040e-b825-4490-bad5-637800da41e2-plugins-config-volume" pod "aa34040e-b825-4490-bad5-637800da41e2" (UID: "aa34040e-b825-4490-bad5-637800da41e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:06:20.454225 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.454204    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa34040e-b825-4490-bad5-637800da41e2-kube-api-access-2px6w" pod "aa34040e-b825-4490-bad5-637800da41e2" (UID: "aa34040e-b825-4490-bad5-637800da41e2"). InnerVolumeSpecName "kube-api-access-2px6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:06:20.455062 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.455036    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-kube-api-access-6xx9v" pod "7b81fac8-afdb-4c6b-a95a-4654a89d9b31" (UID: "7b81fac8-afdb-4c6b-a95a-4654a89d9b31"). InnerVolumeSpecName "kube-api-access-6xx9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:06:20.552114 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.551985    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:20.552114 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.552019    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2px6w\" (UniqueName: \"kubernetes.io/projected/aa34040e-b825-4490-bad5-637800da41e2-kube-api-access-2px6w\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:20.552114 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.552034    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa34040e-b825-4490-bad5-637800da41e2-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:20.552114 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.552048    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xx9v\" (UniqueName: \"kubernetes.io/projected/7b81fac8-afdb-4c6b-a95a-4654a89d9b31-kube-api-access-6xx9v\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:20.638597 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:20.638559    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"]
Oct 30 05:06:21.129470 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:21.129438    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"
Oct 30 05:06:21.129940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:21.129465    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45" event={"ID":"7b81fac8-afdb-4c6b-a95a-4654a89d9b31","Type":"ContainerDied","Data":"e7333167e3b31186dc3d6beb3f7e4f6e0048abf2278597813c8a99f2cf31f73a"}
Oct 30 05:06:21.129940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:21.129543    5072 scope.go:122] "RemoveContainer" containerID="2399637d24a925a3a9b640708d700cca0f44a7923f6bfbec0f9bb21fbdc42334"
Oct 30 05:06:21.131352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:21.131326    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf" event={"ID":"aa34040e-b825-4490-bad5-637800da41e2","Type":"ContainerDied","Data":"9f70b77b644b802e2942a87162a7799bdd8d242d5571ecd1cdb68cfe008b6ce1"}
Oct 30 05:06:21.131518 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:21.131423    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"
Oct 30 05:06:21.144361 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:21.144336    5072 scope.go:122] "RemoveContainer" containerID="f735aa6bf10267b66dc3fc63b04555b0935e28f53936a4c6eccd0b9ff5eafe29"
Oct 30 05:06:21.201731 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:21.201685    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"]
Oct 30 05:06:21.208203 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:21.206400    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7700a7-8b65dd5ff-j66bf"]
Oct 30 05:06:21.241291 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:21.241243    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"]
Oct 30 05:06:21.245775 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:21.245746    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7700a7-79d494ff4d-qvf45"]
Oct 30 05:06:22.410047 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:22.410016    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b81fac8-afdb-4c6b-a95a-4654a89d9b31" path="/var/lib/kubelet/pods/7b81fac8-afdb-4c6b-a95a-4654a89d9b31/volumes"
Oct 30 05:06:22.410471 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:22.410456    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa34040e-b825-4490-bad5-637800da41e2" path="/var/lib/kubelet/pods/aa34040e-b825-4490-bad5-637800da41e2/volumes"
Oct 30 05:06:26.548309 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:26.548278    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"]
Oct 30 05:06:26.582308 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:26.582282    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"]
Oct 30 05:06:28.264639 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:28.264602    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"]
Oct 30 05:06:28.265224 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:28.264854    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerName="istio-proxy" containerID="cri-o://3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a" gracePeriod=30
Oct 30 05:06:31.548800 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:31.548748    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85" podUID="9ad68684-c8f0-4656-9bde-67293af60323" containerName="payload-processing" containerID="cri-o://35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897" gracePeriod=25
Oct 30 05:06:31.582972 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:31.582928    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv" podUID="2e3a563c-81d1-41a3-9c9a-ffe52517fd03" containerName="payload-pre-processing" containerID="cri-o://3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98" gracePeriod=25
Oct 30 05:06:32.520390 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:32.520354    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv" podUID="2e3a563c-81d1-41a3-9c9a-ffe52517fd03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:06:32.520390 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:32.520375    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85" podUID="9ad68684-c8f0-4656-9bde-67293af60323" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:06:33.558831 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.558803    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 05:06:33.647110 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.647037    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-certs\") pod \"1a006882-7f2d-4624-9bab-4d3623a0b75e\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") "
Oct 30 05:06:33.647323 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.647209    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-token\" (UniqueName: \"kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-token\") pod \"1a006882-7f2d-4624-9bab-4d3623a0b75e\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") "
Oct 30 05:06:33.647323 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.647280    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-podinfo\") pod \"1a006882-7f2d-4624-9bab-4d3623a0b75e\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") "
Oct 30 05:06:33.647460 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.647401    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-data\") pod \"1a006882-7f2d-4624-9bab-4d3623a0b75e\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") "
Oct 30 05:06:33.647460 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.647454    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-kube-api-access-ztr7p\" (UniqueName: \"kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-kube-api-access-ztr7p\") pod \"1a006882-7f2d-4624-9bab-4d3623a0b75e\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") "
Oct 30 05:06:33.647582 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.647528    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-credential-socket\") pod \"1a006882-7f2d-4624-9bab-4d3623a0b75e\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") "
Oct 30 05:06:33.647642 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.647584    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a006882-7f2d-4624-9bab-4d3623a0b75e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a006882-7f2d-4624-9bab-4d3623a0b75e-istiod-ca-cert\") pod \"1a006882-7f2d-4624-9bab-4d3623a0b75e\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") "
Oct 30 05:06:33.647642 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.647615    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-envoy\") pod \"1a006882-7f2d-4624-9bab-4d3623a0b75e\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") "
Oct 30 05:06:33.647757 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.647630    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-certs" pod "1a006882-7f2d-4624-9bab-4d3623a0b75e" (UID: "1a006882-7f2d-4624-9bab-4d3623a0b75e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:33.647757 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.647666    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-socket\") pod \"1a006882-7f2d-4624-9bab-4d3623a0b75e\" (UID: \"1a006882-7f2d-4624-9bab-4d3623a0b75e\") "
Oct 30 05:06:33.648304 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.648280    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:33.648765 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.648717    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-socket" pod "1a006882-7f2d-4624-9bab-4d3623a0b75e" (UID: "1a006882-7f2d-4624-9bab-4d3623a0b75e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:33.648871 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.648782    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-credential-socket" pod "1a006882-7f2d-4624-9bab-4d3623a0b75e" (UID: "1a006882-7f2d-4624-9bab-4d3623a0b75e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:33.649054 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.648929    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a006882-7f2d-4624-9bab-4d3623a0b75e-istiod-ca-cert" pod "1a006882-7f2d-4624-9bab-4d3623a0b75e" (UID: "1a006882-7f2d-4624-9bab-4d3623a0b75e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:06:33.649054 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.648985    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-data" pod "1a006882-7f2d-4624-9bab-4d3623a0b75e" (UID: "1a006882-7f2d-4624-9bab-4d3623a0b75e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:33.651262 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.651238    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-kube-api-access-ztr7p" pod "1a006882-7f2d-4624-9bab-4d3623a0b75e" (UID: "1a006882-7f2d-4624-9bab-4d3623a0b75e"). InnerVolumeSpecName "kube-api-access-ztr7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:06:33.651478 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.651456    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-podinfo" pod "1a006882-7f2d-4624-9bab-4d3623a0b75e" (UID: "1a006882-7f2d-4624-9bab-4d3623a0b75e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:06:33.651646 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.651627    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-envoy" pod "1a006882-7f2d-4624-9bab-4d3623a0b75e" (UID: "1a006882-7f2d-4624-9bab-4d3623a0b75e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:33.651761 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.651746    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-token" pod "1a006882-7f2d-4624-9bab-4d3623a0b75e" (UID: "1a006882-7f2d-4624-9bab-4d3623a0b75e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:06:33.749474 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.749437    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:33.749474 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.749467    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:33.749474 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.749476    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:33.749474 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.749484    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztr7p\" (UniqueName: \"kubernetes.io/projected/1a006882-7f2d-4624-9bab-4d3623a0b75e-kube-api-access-ztr7p\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:33.749751 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.749494    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:33.749751 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.749503    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a006882-7f2d-4624-9bab-4d3623a0b75e-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:33.749751 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.749512    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:33.749751 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:33.749522    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a006882-7f2d-4624-9bab-4d3623a0b75e-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:34.255625 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.255592    5072 generic.go:356] "Generic (PLEG): container finished" podID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerID="3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a" exitCode=0
Oct 30 05:06:34.255792 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.255642    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"
Oct 30 05:06:34.255792 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.255669    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" event={"ID":"1a006882-7f2d-4624-9bab-4d3623a0b75e","Type":"ContainerDied","Data":"3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a"}
Oct 30 05:06:34.255792 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.255717    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556" event={"ID":"1a006882-7f2d-4624-9bab-4d3623a0b75e","Type":"ContainerDied","Data":"982e803d449f1cd12e5380c4ff3286d126f87d8a1b20140b2376d8a56c82de4e"}
Oct 30 05:06:34.255792 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.255740    5072 scope.go:122] "RemoveContainer" containerID="3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a"
Oct 30 05:06:34.268319 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.268300    5072 scope.go:122] "RemoveContainer" containerID="764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501"
Oct 30 05:06:34.279727 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.279702    5072 scope.go:122] "RemoveContainer" containerID="3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a"
Oct 30 05:06:34.280042 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:06:34.280019    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a\": container with ID starting with 3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a not found: ID does not exist" containerID="3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a"
Oct 30 05:06:34.280156 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.280055    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a"} err="failed to get container status \"3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a\": rpc error: code = NotFound desc = could not find container \"3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a\": container with ID starting with 3b19c0b50d16ed981f55f5df2f7b1e0d6fa5b2db11efc63ab3a712d9b4dd921a not found: ID does not exist"
Oct 30 05:06:34.280156 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.280097    5072 scope.go:122] "RemoveContainer" containerID="764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501"
Oct 30 05:06:34.280384 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:06:34.280367    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501\": container with ID starting with 764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501 not found: ID does not exist" containerID="764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501"
Oct 30 05:06:34.280458 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.280392    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501"} err="failed to get container status \"764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501\": rpc error: code = NotFound desc = could not find container \"764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501\": container with ID starting with 764bd3047c37ac956be0674df9ea3fa9f2a9c35c10e7e92703dfa1c9cb368501 not found: ID does not exist"
Oct 30 05:06:34.324024 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.323988    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"]
Oct 30 05:06:34.336680 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.336632    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7700a7-openshift-default-dd98dcf55-ns556"]
Oct 30 05:06:34.409652 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:34.409619    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" path="/var/lib/kubelet/pods/1a006882-7f2d-4624-9bab-4d3623a0b75e/volumes"
Oct 30 05:06:35.336935 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:35.336899    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"]
Oct 30 05:06:35.373741 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:35.373705    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"]
Oct 30 05:06:35.662189 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:35.662063    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl" podUID="4bf3c58a-bfdf-4225-9cc5-90f6e10d5639" containerName="main" containerID="cri-o://7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415" gracePeriod=45
Oct 30 05:06:36.016411 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.016357    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:06:36.086055 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.086010    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-dshm\") pod \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") "
Oct 30 05:06:36.086257 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.086151    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-home\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-home\") pod \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") "
Oct 30 05:06:36.086257 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.086220    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tmp-dir\") pod \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") "
Oct 30 05:06:36.086364 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.086257    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-kube-api-access-gwj92\" (UniqueName: \"kubernetes.io/projected/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-kube-api-access-gwj92\") pod \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") "
Oct 30 05:06:36.086403 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.086361    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-model-cache\") pod \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") "
Oct 30 05:06:36.086518 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.086502    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tls-certs\" (UniqueName: \"kubernetes.io/secret/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tls-certs\") pod \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\" (UID: \"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639\") "
Oct 30 05:06:36.087221 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.086847    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tmp-dir" pod "4bf3c58a-bfdf-4225-9cc5-90f6e10d5639" (UID: "4bf3c58a-bfdf-4225-9cc5-90f6e10d5639"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:36.087221 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.086984    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-model-cache" pod "4bf3c58a-bfdf-4225-9cc5-90f6e10d5639" (UID: "4bf3c58a-bfdf-4225-9cc5-90f6e10d5639"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:36.087221 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.087181    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:36.087221 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.087200    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:36.087649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.087329    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-home" pod "4bf3c58a-bfdf-4225-9cc5-90f6e10d5639" (UID: "4bf3c58a-bfdf-4225-9cc5-90f6e10d5639"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:36.089713 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.089671    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-kube-api-access-gwj92" pod "4bf3c58a-bfdf-4225-9cc5-90f6e10d5639" (UID: "4bf3c58a-bfdf-4225-9cc5-90f6e10d5639"). InnerVolumeSpecName "kube-api-access-gwj92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:06:36.089987 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.089952    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tls-certs" pod "4bf3c58a-bfdf-4225-9cc5-90f6e10d5639" (UID: "4bf3c58a-bfdf-4225-9cc5-90f6e10d5639"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:06:36.091746 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.091722    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-dshm" pod "4bf3c58a-bfdf-4225-9cc5-90f6e10d5639" (UID: "4bf3c58a-bfdf-4225-9cc5-90f6e10d5639"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:36.189045 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.189014    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:36.189045 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.189041    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:36.189045 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.189049    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:36.189323 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.189058    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwj92\" (UniqueName: \"kubernetes.io/projected/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639-kube-api-access-gwj92\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:36.273568 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.273479    5072 generic.go:356] "Generic (PLEG): container finished" podID="4bf3c58a-bfdf-4225-9cc5-90f6e10d5639" containerID="7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415" exitCode=0
Oct 30 05:06:36.273568 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.273525    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"
Oct 30 05:06:36.273768 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.273565    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl" event={"ID":"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639","Type":"ContainerDied","Data":"7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415"}
Oct 30 05:06:36.273768 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.273608    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl" event={"ID":"4bf3c58a-bfdf-4225-9cc5-90f6e10d5639","Type":"ContainerDied","Data":"07e70cce0c5960173272ca8809d99e43b3a5a45770fad37fa83e5bd18ce508cc"}
Oct 30 05:06:36.273768 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.273626    5072 scope.go:122] "RemoveContainer" containerID="7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415"
Oct 30 05:06:36.286054 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.286033    5072 scope.go:122] "RemoveContainer" containerID="7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415"
Oct 30 05:06:36.286386 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:06:36.286364    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415\": container with ID starting with 7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415 not found: ID does not exist" containerID="7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415"
Oct 30 05:06:36.286455 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.286398    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415"} err="failed to get container status \"7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415\": rpc error: code = NotFound desc = could not find container \"7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415\": container with ID starting with 7a6d0f4d66f75e16680589c4bb15d3fe2abde417e315090894dd25e83c764415 not found: ID does not exist"
Oct 30 05:06:36.328563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.328529    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"]
Oct 30 05:06:36.341139 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.341108    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-26b72f/test-model-26b72f-kserve-7f9878c5d5-nj8gl"]
Oct 30 05:06:36.412549 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:36.412508    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf3c58a-bfdf-4225-9cc5-90f6e10d5639" path="/var/lib/kubelet/pods/4bf3c58a-bfdf-4225-9cc5-90f6e10d5639/volumes"
Oct 30 05:06:40.337833 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:40.337746    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh" podUID="95d1b5c8-fa40-4c9b-8a93-c29c13595d3a" containerName="payload-processing" containerID="cri-o://f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14" gracePeriod=25
Oct 30 05:06:40.374610 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:40.374573    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v" podUID="87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282" containerName="payload-pre-processing" containerID="cri-o://16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b" gracePeriod=25
Oct 30 05:06:40.617216 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:40.617186    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"
Oct 30 05:06:40.644816 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:40.644783    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-kube-api-access-sj7pk\" (UniqueName: \"kubernetes.io/projected/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-kube-api-access-sj7pk\") pod \"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a\" (UID: \"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a\") "
Oct 30 05:06:40.645009 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:40.644831    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-plugins-config-volume\") pod \"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a\" (UID: \"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a\") "
Oct 30 05:06:40.645285 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:40.645262    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-plugins-config-volume" pod "95d1b5c8-fa40-4c9b-8a93-c29c13595d3a" (UID: "95d1b5c8-fa40-4c9b-8a93-c29c13595d3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:06:40.645413 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:40.645401    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:40.647283 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:40.647258    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-kube-api-access-sj7pk" pod "95d1b5c8-fa40-4c9b-8a93-c29c13595d3a" (UID: "95d1b5c8-fa40-4c9b-8a93-c29c13595d3a"). InnerVolumeSpecName "kube-api-access-sj7pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:06:40.746557 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:40.746519    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj7pk\" (UniqueName: \"kubernetes.io/projected/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a-kube-api-access-sj7pk\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:41.312877 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:41.312843    5072 generic.go:356] "Generic (PLEG): container finished" podID="95d1b5c8-fa40-4c9b-8a93-c29c13595d3a" containerID="f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14" exitCode=0
Oct 30 05:06:41.313065 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:41.312894    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"
Oct 30 05:06:41.313065 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:41.312923    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh" event={"ID":"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a","Type":"ContainerDied","Data":"f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14"}
Oct 30 05:06:41.313065 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:41.312983    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh" event={"ID":"95d1b5c8-fa40-4c9b-8a93-c29c13595d3a","Type":"ContainerDied","Data":"295ba4f21afaa137101459e692142f54591661bf79c7e23bf6b789a95beb68c7"}
Oct 30 05:06:41.313065 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:41.313011    5072 scope.go:122] "RemoveContainer" containerID="f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14"
Oct 30 05:06:41.325484 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:41.325462    5072 scope.go:122] "RemoveContainer" containerID="f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14"
Oct 30 05:06:41.325788 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:06:41.325763    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14\": container with ID starting with f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14 not found: ID does not exist" containerID="f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14"
Oct 30 05:06:41.325838 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:41.325798    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14"} err="failed to get container status \"f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14\": rpc error: code = NotFound desc = could not find container \"f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14\": container with ID starting with f6b6eb092045b08cb3b5c8696889abb9d982aa768a88126aabe2bd090c4a5e14 not found: ID does not exist"
Oct 30 05:06:41.368047 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:41.368010    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"]
Oct 30 05:06:41.382922 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:41.382887    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e592d7-846f4f9454-glsqh"]
Oct 30 05:06:41.753063 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:06:41.753018    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:06:42.410062 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:42.410030    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d1b5c8-fa40-4c9b-8a93-c29c13595d3a" path="/var/lib/kubelet/pods/95d1b5c8-fa40-4c9b-8a93-c29c13595d3a/volumes"
Oct 30 05:06:42.520746 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:42.520700    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85" podUID="9ad68684-c8f0-4656-9bde-67293af60323" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:06:42.520746 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:42.520727    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv" podUID="2e3a563c-81d1-41a3-9c9a-ffe52517fd03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:06:48.310724 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:48.310685    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v" podUID="87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:06:51.913925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:51.913896    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"
Oct 30 05:06:51.975337 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:51.974153    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"
Oct 30 05:06:52.013821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.013785    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad68684-c8f0-4656-9bde-67293af60323-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad68684-c8f0-4656-9bde-67293af60323-plugins-config-volume\") pod \"9ad68684-c8f0-4656-9bde-67293af60323\" (UID: \"9ad68684-c8f0-4656-9bde-67293af60323\") "
Oct 30 05:06:52.013991 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.013834    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad68684-c8f0-4656-9bde-67293af60323-kube-api-access-ttrbd\" (UniqueName: \"kubernetes.io/projected/9ad68684-c8f0-4656-9bde-67293af60323-kube-api-access-ttrbd\") pod \"9ad68684-c8f0-4656-9bde-67293af60323\" (UID: \"9ad68684-c8f0-4656-9bde-67293af60323\") "
Oct 30 05:06:52.014245 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.014206    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad68684-c8f0-4656-9bde-67293af60323-plugins-config-volume" pod "9ad68684-c8f0-4656-9bde-67293af60323" (UID: "9ad68684-c8f0-4656-9bde-67293af60323"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:06:52.017551 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.017515    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad68684-c8f0-4656-9bde-67293af60323-kube-api-access-ttrbd" pod "9ad68684-c8f0-4656-9bde-67293af60323" (UID: "9ad68684-c8f0-4656-9bde-67293af60323"). InnerVolumeSpecName "kube-api-access-ttrbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:06:52.114742 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.114647    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-plugins-config-volume\") pod \"2e3a563c-81d1-41a3-9c9a-ffe52517fd03\" (UID: \"2e3a563c-81d1-41a3-9c9a-ffe52517fd03\") "
Oct 30 05:06:52.114913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.114890    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-kube-api-access-blp4w\" (UniqueName: \"kubernetes.io/projected/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-kube-api-access-blp4w\") pod \"2e3a563c-81d1-41a3-9c9a-ffe52517fd03\" (UID: \"2e3a563c-81d1-41a3-9c9a-ffe52517fd03\") "
Oct 30 05:06:52.115126 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.115058    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-plugins-config-volume" pod "2e3a563c-81d1-41a3-9c9a-ffe52517fd03" (UID: "2e3a563c-81d1-41a3-9c9a-ffe52517fd03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:06:52.115449 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.115422    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:52.115559 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.115453    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad68684-c8f0-4656-9bde-67293af60323-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:52.115559 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.115471    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttrbd\" (UniqueName: \"kubernetes.io/projected/9ad68684-c8f0-4656-9bde-67293af60323-kube-api-access-ttrbd\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:52.117593 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.117554    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-kube-api-access-blp4w" pod "2e3a563c-81d1-41a3-9c9a-ffe52517fd03" (UID: "2e3a563c-81d1-41a3-9c9a-ffe52517fd03"). InnerVolumeSpecName "kube-api-access-blp4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:06:52.216787 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.216746    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blp4w\" (UniqueName: \"kubernetes.io/projected/2e3a563c-81d1-41a3-9c9a-ffe52517fd03-kube-api-access-blp4w\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:06:52.408421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.408336    5072 generic.go:356] "Generic (PLEG): container finished" podID="2e3a563c-81d1-41a3-9c9a-ffe52517fd03" containerID="3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98" exitCode=0
Oct 30 05:06:52.408619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.408449    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"
Oct 30 05:06:52.410038 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.410010    5072 generic.go:356] "Generic (PLEG): container finished" podID="9ad68684-c8f0-4656-9bde-67293af60323" containerID="35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897" exitCode=0
Oct 30 05:06:52.410179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.410139    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"
Oct 30 05:06:52.410862 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.410841    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv" event={"ID":"2e3a563c-81d1-41a3-9c9a-ffe52517fd03","Type":"ContainerDied","Data":"3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98"}
Oct 30 05:06:52.410926 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.410882    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv" event={"ID":"2e3a563c-81d1-41a3-9c9a-ffe52517fd03","Type":"ContainerDied","Data":"a7a8bebabc24c1987d0569e3a7e68638a7ba8020a667fc6ecfcdf5570866577f"}
Oct 30 05:06:52.410926 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.410905    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85" event={"ID":"9ad68684-c8f0-4656-9bde-67293af60323","Type":"ContainerDied","Data":"35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897"}
Oct 30 05:06:52.410926 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.410922    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85" event={"ID":"9ad68684-c8f0-4656-9bde-67293af60323","Type":"ContainerDied","Data":"f7134d586bef8c887ea5382f48c47c8369e5af1f20af600b25017696d2abd339"}
Oct 30 05:06:52.411038 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.410941    5072 scope.go:122] "RemoveContainer" containerID="3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98"
Oct 30 05:06:52.423349 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.423322    5072 scope.go:122] "RemoveContainer" containerID="3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98"
Oct 30 05:06:52.423648 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:06:52.423625    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98\": container with ID starting with 3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98 not found: ID does not exist" containerID="3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98"
Oct 30 05:06:52.423745 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.423659    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98"} err="failed to get container status \"3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98\": rpc error: code = NotFound desc = could not find container \"3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98\": container with ID starting with 3e049cc79bcadf3045e98494345d48fe84aad26635549538adf14d0cfe87cc98 not found: ID does not exist"
Oct 30 05:06:52.423745 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.423683    5072 scope.go:122] "RemoveContainer" containerID="35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897"
Oct 30 05:06:52.434031 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.434013    5072 scope.go:122] "RemoveContainer" containerID="35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897"
Oct 30 05:06:52.434320 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:06:52.434300    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897\": container with ID starting with 35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897 not found: ID does not exist" containerID="35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897"
Oct 30 05:06:52.434404 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.434332    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897"} err="failed to get container status \"35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897\": rpc error: code = NotFound desc = could not find container \"35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897\": container with ID starting with 35fd9c069b90ccad6e8ecf1a0f146ccc8c3c6513a0eb4a1638a4bc9090401897 not found: ID does not exist"
Oct 30 05:06:52.478157 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.478127    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"]
Oct 30 05:06:52.481534 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.481513    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9f158a-87c6d57f5-bmdkv"]
Oct 30 05:06:52.516783 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.516747    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"]
Oct 30 05:06:52.520352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:52.520326    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9f158a-bf66cb696-vpp85"]
Oct 30 05:06:54.409798 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:54.409764    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3a563c-81d1-41a3-9c9a-ffe52517fd03" path="/var/lib/kubelet/pods/2e3a563c-81d1-41a3-9c9a-ffe52517fd03/volumes"
Oct 30 05:06:54.410285 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:54.410269    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad68684-c8f0-4656-9bde-67293af60323" path="/var/lib/kubelet/pods/9ad68684-c8f0-4656-9bde-67293af60323/volumes"
Oct 30 05:06:58.310731 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:06:58.310695    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v" podUID="87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:07:00.655647 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:00.655624    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"
Oct 30 05:07:00.716419 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:00.716379    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-kube-api-access-swgpm\" (UniqueName: \"kubernetes.io/projected/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-kube-api-access-swgpm\") pod \"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282\" (UID: \"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282\") "
Oct 30 05:07:00.716592 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:00.716546    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-plugins-config-volume\") pod \"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282\" (UID: \"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282\") "
Oct 30 05:07:00.716948 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:00.716921    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-plugins-config-volume" pod "87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282" (UID: "87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:07:00.717188 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:00.717169    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:07:00.718743 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:00.718718    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-kube-api-access-swgpm" pod "87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282" (UID: "87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282"). InnerVolumeSpecName "kube-api-access-swgpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:07:00.818668 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:00.818638    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swgpm\" (UniqueName: \"kubernetes.io/projected/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282-kube-api-access-swgpm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:07:01.057861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:01.057828    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:07:01.076378 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:01.076353    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:07:01.501999 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:01.501965    5072 generic.go:356] "Generic (PLEG): container finished" podID="87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282" containerID="16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b" exitCode=0
Oct 30 05:07:01.502346 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:01.502035    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"
Oct 30 05:07:01.502505 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:01.502053    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v" event={"ID":"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282","Type":"ContainerDied","Data":"16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b"}
Oct 30 05:07:01.502505 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:01.502481    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v" event={"ID":"87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282","Type":"ContainerDied","Data":"c9cb6a2141de9117e506a4231da0bf406669e7e6006c552bbe7d58dcdfdf9c95"}
Oct 30 05:07:01.502631 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:01.502515    5072 scope.go:122] "RemoveContainer" containerID="16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b"
Oct 30 05:07:01.516350 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:01.516325    5072 scope.go:122] "RemoveContainer" containerID="16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b"
Oct 30 05:07:01.516699 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:07:01.516673    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b\": container with ID starting with 16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b not found: ID does not exist" containerID="16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b"
Oct 30 05:07:01.516781 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:01.516710    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b"} err="failed to get container status \"16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b\": rpc error: code = NotFound desc = could not find container \"16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b\": container with ID starting with 16ba2376f3ecda2c9ce4db8b40a596c99bd70fa988951ecf79d2509f57e47c4b not found: ID does not exist"
Oct 30 05:07:01.559735 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:01.559706    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"]
Oct 30 05:07:01.566020 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:01.565993    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e592d7-799cf664d7-gdr2v"]
Oct 30 05:07:02.419058 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:02.419019    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282" path="/var/lib/kubelet/pods/87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282/volumes"
Oct 30 05:07:02.939709 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:02.939677    5072 scope.go:122] "RemoveContainer" containerID="804cbdd3632ea7518375c56f15951e639f1252de907a7a6aed06f35ef3560d2c"
Oct 30 05:07:02.952228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:02.951947    5072 scope.go:122] "RemoveContainer" containerID="bcc4b50be5f3aae4dbc464e55bfee70cce5268b2fe09f70572176181fc8ecaa2"
Oct 30 05:07:11.753363 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:07:11.753321    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:07:14.389143 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:14.389107    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"]
Oct 30 05:07:19.389957 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:19.389914    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm" podUID="934a686e-a82d-403e-b13f-97637b888cee" containerName="payload-processing" containerID="cri-o://49c727f733b38bb310472c23b22e402c0668f114e443b99407bf893eddbb7cf6" gracePeriod=25
Oct 30 05:07:19.663457 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:19.663412    5072 generic.go:356] "Generic (PLEG): container finished" podID="934a686e-a82d-403e-b13f-97637b888cee" containerID="49c727f733b38bb310472c23b22e402c0668f114e443b99407bf893eddbb7cf6" exitCode=0
Oct 30 05:07:19.663590 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:19.663457    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm" event={"ID":"934a686e-a82d-403e-b13f-97637b888cee","Type":"ContainerDied","Data":"49c727f733b38bb310472c23b22e402c0668f114e443b99407bf893eddbb7cf6"}
Oct 30 05:07:19.663590 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:19.663511    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm" event={"ID":"934a686e-a82d-403e-b13f-97637b888cee","Type":"ContainerDied","Data":"dc7b8c124b92f860e8cada5a0b9860b75c85979d8cf0f5c62ecf2c843c9e47a7"}
Oct 30 05:07:19.663590 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:19.663532    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc7b8c124b92f860e8cada5a0b9860b75c85979d8cf0f5c62ecf2c843c9e47a7"
Oct 30 05:07:19.684506 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:19.684480    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"
Oct 30 05:07:19.775402 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:19.775361    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/934a686e-a82d-403e-b13f-97637b888cee-kube-api-access-pblfh\" (UniqueName: \"kubernetes.io/projected/934a686e-a82d-403e-b13f-97637b888cee-kube-api-access-pblfh\") pod \"934a686e-a82d-403e-b13f-97637b888cee\" (UID: \"934a686e-a82d-403e-b13f-97637b888cee\") "
Oct 30 05:07:19.775600 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:19.775425    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/934a686e-a82d-403e-b13f-97637b888cee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934a686e-a82d-403e-b13f-97637b888cee-plugins-config-volume\") pod \"934a686e-a82d-403e-b13f-97637b888cee\" (UID: \"934a686e-a82d-403e-b13f-97637b888cee\") "
Oct 30 05:07:19.776212 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:19.776174    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934a686e-a82d-403e-b13f-97637b888cee-plugins-config-volume" pod "934a686e-a82d-403e-b13f-97637b888cee" (UID: "934a686e-a82d-403e-b13f-97637b888cee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:07:19.778463 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:19.778418    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934a686e-a82d-403e-b13f-97637b888cee-kube-api-access-pblfh" pod "934a686e-a82d-403e-b13f-97637b888cee" (UID: "934a686e-a82d-403e-b13f-97637b888cee"). InnerVolumeSpecName "kube-api-access-pblfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:07:19.877744 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:19.877707    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pblfh\" (UniqueName: \"kubernetes.io/projected/934a686e-a82d-403e-b13f-97637b888cee-kube-api-access-pblfh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:07:19.877744 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:19.877740    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934a686e-a82d-403e-b13f-97637b888cee-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:07:20.670729 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:20.670653    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"
Oct 30 05:07:20.726332 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:20.726301    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"]
Oct 30 05:07:20.738559 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:20.738532    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-26b72f-8499db668d-zkxrm"]
Oct 30 05:07:22.410226 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:22.410195    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="934a686e-a82d-403e-b13f-97637b888cee" path="/var/lib/kubelet/pods/934a686e-a82d-403e-b13f-97637b888cee/volumes"
Oct 30 05:07:29.340745 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.340707    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"]
Oct 30 05:07:29.341454 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341438    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2e3a563c-81d1-41a3-9c9a-ffe52517fd03" containerName="payload-pre-processing"
Oct 30 05:07:29.341513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341455    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3a563c-81d1-41a3-9c9a-ffe52517fd03" containerName="payload-pre-processing"
Oct 30 05:07:29.341513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341471    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5806f630-df78-471c-a399-e3c6413f370b" containerName="revoke-keys"
Oct 30 05:07:29.341513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341476    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5806f630-df78-471c-a399-e3c6413f370b" containerName="revoke-keys"
Oct 30 05:07:29.341513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341492    5072 cpu_manager.go:414] "containerMap: removing container" podUID="aa34040e-b825-4490-bad5-637800da41e2" containerName="payload-pre-processing"
Oct 30 05:07:29.341513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341497    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa34040e-b825-4490-bad5-637800da41e2" containerName="payload-pre-processing"
Oct 30 05:07:29.341513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341505    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7b81fac8-afdb-4c6b-a95a-4654a89d9b31" containerName="payload-processing"
Oct 30 05:07:29.341513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341511    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b81fac8-afdb-4c6b-a95a-4654a89d9b31" containerName="payload-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341520    5072 cpu_manager.go:414] "containerMap: removing container" podUID="95d1b5c8-fa40-4c9b-8a93-c29c13595d3a" containerName="payload-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341525    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d1b5c8-fa40-4c9b-8a93-c29c13595d3a" containerName="payload-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341532    5072 cpu_manager.go:414] "containerMap: removing container" podUID="934a686e-a82d-403e-b13f-97637b888cee" containerName="payload-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341536    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934a686e-a82d-403e-b13f-97637b888cee" containerName="payload-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341542    5072 cpu_manager.go:414] "containerMap: removing container" podUID="4bf3c58a-bfdf-4225-9cc5-90f6e10d5639" containerName="main"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341546    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf3c58a-bfdf-4225-9cc5-90f6e10d5639" containerName="main"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341561    5072 cpu_manager.go:414] "containerMap: removing container" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerName="istio-proxy"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341565    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerName="istio-proxy"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341574    5072 cpu_manager.go:414] "containerMap: removing container" podUID="9ad68684-c8f0-4656-9bde-67293af60323" containerName="payload-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341578    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad68684-c8f0-4656-9bde-67293af60323" containerName="payload-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341587    5072 cpu_manager.go:414] "containerMap: removing container" podUID="87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282" containerName="payload-pre-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341591    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282" containerName="payload-pre-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341688    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d1b5c8-fa40-4c9b-8a93-c29c13595d3a" containerName="payload-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341696    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="934a686e-a82d-403e-b13f-97637b888cee" containerName="payload-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341702    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf3c58a-bfdf-4225-9cc5-90f6e10d5639" containerName="main"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341711    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerName="istio-proxy"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341717    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad68684-c8f0-4656-9bde-67293af60323" containerName="payload-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341725    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a61dd8-9bd0-4ec2-bffd-0bfb4d03a282" containerName="payload-pre-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341730    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerName="istio-proxy"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341740    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3a563c-81d1-41a3-9c9a-ffe52517fd03" containerName="payload-pre-processing"
Oct 30 05:07:29.341739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341747    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5806f630-df78-471c-a399-e3c6413f370b" containerName="revoke-keys"
Oct 30 05:07:29.342401 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341754    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa34040e-b825-4490-bad5-637800da41e2" containerName="payload-pre-processing"
Oct 30 05:07:29.342401 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.341759    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b81fac8-afdb-4c6b-a95a-4654a89d9b31" containerName="payload-processing"
Oct 30 05:07:29.346723 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.346705    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:29.369523 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.369491    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"]
Oct 30 05:07:29.376799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.376772    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:07:29.376964 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.376870    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:07:29.421408 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.421368    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0674b215-1609-416c-8921-13b00c6ad543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1b971c5-bcthc\" (UID: \"0674b215-1609-416c-8921-13b00c6ad543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:29.421599 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.421508    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltgvb\" (UniqueName: \"kubernetes.io/projected/0674b215-1609-416c-8921-13b00c6ad543-kube-api-access-ltgvb\") pod \"maas-api-revoke-sub-b1b971c5-bcthc\" (UID: \"0674b215-1609-416c-8921-13b00c6ad543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:29.421599 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.421546    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0674b215-1609-416c-8921-13b00c6ad543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1b971c5-bcthc\" (UID: \"0674b215-1609-416c-8921-13b00c6ad543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:29.522652 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.522622    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0674b215-1609-416c-8921-13b00c6ad543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1b971c5-bcthc\" (UID: \"0674b215-1609-416c-8921-13b00c6ad543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:29.522865 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.522715    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgvb\" (UniqueName: \"kubernetes.io/projected/0674b215-1609-416c-8921-13b00c6ad543-kube-api-access-ltgvb\") pod \"maas-api-revoke-sub-b1b971c5-bcthc\" (UID: \"0674b215-1609-416c-8921-13b00c6ad543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:29.522865 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.522741    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0674b215-1609-416c-8921-13b00c6ad543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1b971c5-bcthc\" (UID: \"0674b215-1609-416c-8921-13b00c6ad543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:29.523346 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.523323    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0674b215-1609-416c-8921-13b00c6ad543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1b971c5-bcthc\" (UID: \"0674b215-1609-416c-8921-13b00c6ad543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:29.525370 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.525347    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0674b215-1609-416c-8921-13b00c6ad543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1b971c5-bcthc\" (UID: \"0674b215-1609-416c-8921-13b00c6ad543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:29.546489 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.546461    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgvb\" (UniqueName: \"kubernetes.io/projected/0674b215-1609-416c-8921-13b00c6ad543-kube-api-access-ltgvb\") pod \"maas-api-revoke-sub-b1b971c5-bcthc\" (UID: \"0674b215-1609-416c-8921-13b00c6ad543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:29.660900 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.660829    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:29.905574 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.905539    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"]
Oct 30 05:07:29.906265 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.906250    5072 cpu_manager.go:414] "containerMap: removing container" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerName="istio-proxy"
Oct 30 05:07:29.906315 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.906266    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a006882-7f2d-4624-9bab-4d3623a0b75e" containerName="istio-proxy"
Oct 30 05:07:29.911253 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.911199    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"
Oct 30 05:07:29.926914 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.926887    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b5496290-dockercfg-4chmz\""
Oct 30 05:07:29.927056 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.926927    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b5496290\""
Oct 30 05:07:29.928865 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:29.928841    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"]
Oct 30 05:07:30.030828 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.026302    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"]
Oct 30 05:07:30.030828 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.026376    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"]
Oct 30 05:07:30.052286 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.032093    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb20def-2c1e-4df2-b96a-c743d5c37939-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt\" (UID: \"5cb20def-2c1e-4df2-b96a-c743d5c37939\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"
Oct 30 05:07:30.052286 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.032218    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86x22\" (UniqueName: \"kubernetes.io/projected/5cb20def-2c1e-4df2-b96a-c743d5c37939-kube-api-access-86x22\") pod \"payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt\" (UID: \"5cb20def-2c1e-4df2-b96a-c743d5c37939\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"
Oct 30 05:07:30.052286 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.048256    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"
Oct 30 05:07:30.077896 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.074280    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"]
Oct 30 05:07:30.138786 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.138751    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbbac18-76cc-4c8b-93aa-2346668f419a-plugins-config-volume\") pod \"payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw\" (UID: \"ecbbac18-76cc-4c8b-93aa-2346668f419a\") " pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"
Oct 30 05:07:30.138940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.138911    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb20def-2c1e-4df2-b96a-c743d5c37939-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt\" (UID: \"5cb20def-2c1e-4df2-b96a-c743d5c37939\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"
Oct 30 05:07:30.139003 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.138946    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txd97\" (UniqueName: \"kubernetes.io/projected/ecbbac18-76cc-4c8b-93aa-2346668f419a-kube-api-access-txd97\") pod \"payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw\" (UID: \"ecbbac18-76cc-4c8b-93aa-2346668f419a\") " pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"
Oct 30 05:07:30.139058 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.139013    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86x22\" (UniqueName: \"kubernetes.io/projected/5cb20def-2c1e-4df2-b96a-c743d5c37939-kube-api-access-86x22\") pod \"payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt\" (UID: \"5cb20def-2c1e-4df2-b96a-c743d5c37939\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"
Oct 30 05:07:30.139790 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.139766    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb20def-2c1e-4df2-b96a-c743d5c37939-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt\" (UID: \"5cb20def-2c1e-4df2-b96a-c743d5c37939\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"
Oct 30 05:07:30.162894 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.162842    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86x22\" (UniqueName: \"kubernetes.io/projected/5cb20def-2c1e-4df2-b96a-c743d5c37939-kube-api-access-86x22\") pod \"payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt\" (UID: \"5cb20def-2c1e-4df2-b96a-c743d5c37939\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"
Oct 30 05:07:30.228118 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.228093    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"
Oct 30 05:07:30.240291 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.240267    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbbac18-76cc-4c8b-93aa-2346668f419a-plugins-config-volume\") pod \"payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw\" (UID: \"ecbbac18-76cc-4c8b-93aa-2346668f419a\") " pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"
Oct 30 05:07:30.240448 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.240417    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txd97\" (UniqueName: \"kubernetes.io/projected/ecbbac18-76cc-4c8b-93aa-2346668f419a-kube-api-access-txd97\") pod \"payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw\" (UID: \"ecbbac18-76cc-4c8b-93aa-2346668f419a\") " pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"
Oct 30 05:07:30.241131 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.241109    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbbac18-76cc-4c8b-93aa-2346668f419a-plugins-config-volume\") pod \"payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw\" (UID: \"ecbbac18-76cc-4c8b-93aa-2346668f419a\") " pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"
Oct 30 05:07:30.264707 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.264680    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txd97\" (UniqueName: \"kubernetes.io/projected/ecbbac18-76cc-4c8b-93aa-2346668f419a-kube-api-access-txd97\") pod \"payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw\" (UID: \"ecbbac18-76cc-4c8b-93aa-2346668f419a\") " pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"
Oct 30 05:07:30.403464 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.403440    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"
Oct 30 05:07:30.537374 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:07:30.537336    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb20def_2c1e_4df2_b96a_c743d5c37939.slice/crio-4b64895c9a52ee2b5cac062eacf9d0f836cc568b5680eae2d6a28b9008a8529d WatchSource:0}: Error finding container 4b64895c9a52ee2b5cac062eacf9d0f836cc568b5680eae2d6a28b9008a8529d: Status 404 returned error can't find the container with id 4b64895c9a52ee2b5cac062eacf9d0f836cc568b5680eae2d6a28b9008a8529d
Oct 30 05:07:30.538145 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.538116    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"]
Oct 30 05:07:30.708672 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:07:30.708640    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbbac18_76cc_4c8b_93aa_2346668f419a.slice/crio-984e6e7d390d5c59e3ecb08e8e2c854faa82c286a5c0b220a093aa67df7e53a3 WatchSource:0}: Error finding container 984e6e7d390d5c59e3ecb08e8e2c854faa82c286a5c0b220a093aa67df7e53a3: Status 404 returned error can't find the container with id 984e6e7d390d5c59e3ecb08e8e2c854faa82c286a5c0b220a093aa67df7e53a3
Oct 30 05:07:30.708809 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.708737    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"]
Oct 30 05:07:30.763585 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.763553    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw" event={"ID":"ecbbac18-76cc-4c8b-93aa-2346668f419a","Type":"ContainerStarted","Data":"984e6e7d390d5c59e3ecb08e8e2c854faa82c286a5c0b220a093aa67df7e53a3"}
Oct 30 05:07:30.765376 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.765345    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc" event={"ID":"0674b215-1609-416c-8921-13b00c6ad543","Type":"ContainerStarted","Data":"aef5e6adce6565067f248fa3d4d96ab9ecd28f0bff30ae71265a39f0efc930df"}
Oct 30 05:07:30.765497 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.765400    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc" event={"ID":"0674b215-1609-416c-8921-13b00c6ad543","Type":"ContainerStarted","Data":"c7a11547803144068f17c54941f5d8e911a0e084bd3ec5fa3a5c1aff2f33cc79"}
Oct 30 05:07:30.766979 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.766956    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt" event={"ID":"5cb20def-2c1e-4df2-b96a-c743d5c37939","Type":"ContainerStarted","Data":"8ba3864a0c9fba27c0604f422e7f5c78af617c7b3ed80c310c68072798faa1d4"}
Oct 30 05:07:30.767105 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.766997    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt" event={"ID":"5cb20def-2c1e-4df2-b96a-c743d5c37939","Type":"ContainerStarted","Data":"4b64895c9a52ee2b5cac062eacf9d0f836cc568b5680eae2d6a28b9008a8529d"}
Oct 30 05:07:30.767105 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.767094    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"
Oct 30 05:07:30.807164 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.807122    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc" podStartSLOduration=1.807108268 podStartE2EDuration="1.807108268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:30.805983048 +0000 UTC m=+4230.547406147" watchObservedRunningTime="2026-10-30 05:07:30.807108268 +0000 UTC m=+4230.548531365"
Oct 30 05:07:30.854718 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:30.854655    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt" podStartSLOduration=1.854642001 podStartE2EDuration="1.854642001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:30.85103282 +0000 UTC m=+4230.592455943" watchObservedRunningTime="2026-10-30 05:07:30.854642001 +0000 UTC m=+4230.596065134"
Oct 30 05:07:31.780202 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:31.780167    5072 generic.go:356] "Generic (PLEG): container finished" podID="0674b215-1609-416c-8921-13b00c6ad543" containerID="aef5e6adce6565067f248fa3d4d96ab9ecd28f0bff30ae71265a39f0efc930df" exitCode=0
Oct 30 05:07:31.780689 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:31.780260    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc" event={"ID":"0674b215-1609-416c-8921-13b00c6ad543","Type":"ContainerDied","Data":"aef5e6adce6565067f248fa3d4d96ab9ecd28f0bff30ae71265a39f0efc930df"}
Oct 30 05:07:31.782132 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:31.782109    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw" event={"ID":"ecbbac18-76cc-4c8b-93aa-2346668f419a","Type":"ContainerStarted","Data":"694f26abe4dea62c8a1b6be425276949e3f181322712ef6675700a741391e614"}
Oct 30 05:07:31.782293 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:31.782163    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"
Oct 30 05:07:31.864866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:31.864751    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw" podStartSLOduration=2.864731068 podStartE2EDuration="2.864731068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:31.864272946 +0000 UTC m=+4231.605696046" watchObservedRunningTime="2026-10-30 05:07:31.864731068 +0000 UTC m=+4231.606154168"
Oct 30 05:07:32.971566 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:32.971543    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:33.083832 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:33.083758    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0674b215-1609-416c-8921-13b00c6ad543-kube-api-access-ltgvb\" (UniqueName: \"kubernetes.io/projected/0674b215-1609-416c-8921-13b00c6ad543-kube-api-access-ltgvb\") pod \"0674b215-1609-416c-8921-13b00c6ad543\" (UID: \"0674b215-1609-416c-8921-13b00c6ad543\") "
Oct 30 05:07:33.083832 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:33.083800    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0674b215-1609-416c-8921-13b00c6ad543-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0674b215-1609-416c-8921-13b00c6ad543-maas-api-openshift-ca\") pod \"0674b215-1609-416c-8921-13b00c6ad543\" (UID: \"0674b215-1609-416c-8921-13b00c6ad543\") "
Oct 30 05:07:33.084023 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:33.083847    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0674b215-1609-416c-8921-13b00c6ad543-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0674b215-1609-416c-8921-13b00c6ad543-maas-api-xks-ca\") pod \"0674b215-1609-416c-8921-13b00c6ad543\" (UID: \"0674b215-1609-416c-8921-13b00c6ad543\") "
Oct 30 05:07:33.084286 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:33.084254    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0674b215-1609-416c-8921-13b00c6ad543-maas-api-openshift-ca" pod "0674b215-1609-416c-8921-13b00c6ad543" (UID: "0674b215-1609-416c-8921-13b00c6ad543"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:07:33.084563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:33.084516    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0674b215-1609-416c-8921-13b00c6ad543-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:07:33.086232 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:33.086205    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0674b215-1609-416c-8921-13b00c6ad543-maas-api-xks-ca" pod "0674b215-1609-416c-8921-13b00c6ad543" (UID: "0674b215-1609-416c-8921-13b00c6ad543"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:07:33.086349 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:33.086311    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0674b215-1609-416c-8921-13b00c6ad543-kube-api-access-ltgvb" pod "0674b215-1609-416c-8921-13b00c6ad543" (UID: "0674b215-1609-416c-8921-13b00c6ad543"). InnerVolumeSpecName "kube-api-access-ltgvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:07:33.186202 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:33.186172    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltgvb\" (UniqueName: \"kubernetes.io/projected/0674b215-1609-416c-8921-13b00c6ad543-kube-api-access-ltgvb\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:07:33.186202 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:33.186199    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0674b215-1609-416c-8921-13b00c6ad543-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:07:33.797594 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:33.797551    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc" event={"ID":"0674b215-1609-416c-8921-13b00c6ad543","Type":"ContainerDied","Data":"c7a11547803144068f17c54941f5d8e911a0e084bd3ec5fa3a5c1aff2f33cc79"}
Oct 30 05:07:33.797594 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:33.797587    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"
Oct 30 05:07:33.797594 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:33.797600    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7a11547803144068f17c54941f5d8e911a0e084bd3ec5fa3a5c1aff2f33cc79"
Oct 30 05:07:41.753028 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:07:41.752988    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:07:41.785333 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:41.785306    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"
Oct 30 05:07:42.792577 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:42.792542    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"
Oct 30 05:07:56.009526 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:56.009475    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"]
Oct 30 05:07:56.017709 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:07:56.010156    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm" podUID="ba3bd8c9-443a-45a3-9965-02c57f42164b" containerName="istio-proxy" containerID="cri-o://c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4" gracePeriod=30
Oct 30 05:08:01.341903 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.341867    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:08:01.474359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.474318    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-token\" (UniqueName: \"kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-token\") pod \"ba3bd8c9-443a-45a3-9965-02c57f42164b\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") "
Oct 30 05:08:01.474550 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.474399    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-envoy\") pod \"ba3bd8c9-443a-45a3-9965-02c57f42164b\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") "
Oct 30 05:08:01.474550 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.474474    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-socket\") pod \"ba3bd8c9-443a-45a3-9965-02c57f42164b\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") "
Oct 30 05:08:01.474689 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.474554    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-credential-socket\") pod \"ba3bd8c9-443a-45a3-9965-02c57f42164b\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") "
Oct 30 05:08:01.474689 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.474592    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-certs\") pod \"ba3bd8c9-443a-45a3-9965-02c57f42164b\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") "
Oct 30 05:08:01.474689 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.474632    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-podinfo\") pod \"ba3bd8c9-443a-45a3-9965-02c57f42164b\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") "
Oct 30 05:08:01.474689 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.474669    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3bd8c9-443a-45a3-9965-02c57f42164b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3bd8c9-443a-45a3-9965-02c57f42164b-istiod-ca-cert\") pod \"ba3bd8c9-443a-45a3-9965-02c57f42164b\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") "
Oct 30 05:08:01.474922 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.474705    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-data\") pod \"ba3bd8c9-443a-45a3-9965-02c57f42164b\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") "
Oct 30 05:08:01.474922 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.474775    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-socket" pod "ba3bd8c9-443a-45a3-9965-02c57f42164b" (UID: "ba3bd8c9-443a-45a3-9965-02c57f42164b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:08:01.474922 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.474836    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-kube-api-access-rt79z\" (UniqueName: \"kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-kube-api-access-rt79z\") pod \"ba3bd8c9-443a-45a3-9965-02c57f42164b\" (UID: \"ba3bd8c9-443a-45a3-9965-02c57f42164b\") "
Oct 30 05:08:01.474922 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.474845    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-credential-socket" pod "ba3bd8c9-443a-45a3-9965-02c57f42164b" (UID: "ba3bd8c9-443a-45a3-9965-02c57f42164b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:08:01.475223 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.475065    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-certs" pod "ba3bd8c9-443a-45a3-9965-02c57f42164b" (UID: "ba3bd8c9-443a-45a3-9965-02c57f42164b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:08:01.475440 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.475415    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3bd8c9-443a-45a3-9965-02c57f42164b-istiod-ca-cert" pod "ba3bd8c9-443a-45a3-9965-02c57f42164b" (UID: "ba3bd8c9-443a-45a3-9965-02c57f42164b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:08:01.475440 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.475432    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.475779 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.475479    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.475779 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.475494    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.475779 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.475436    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-data" pod "ba3bd8c9-443a-45a3-9965-02c57f42164b" (UID: "ba3bd8c9-443a-45a3-9965-02c57f42164b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:08:01.476983 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.476952    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-token" pod "ba3bd8c9-443a-45a3-9965-02c57f42164b" (UID: "ba3bd8c9-443a-45a3-9965-02c57f42164b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:08:01.477247 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.477229    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-envoy" pod "ba3bd8c9-443a-45a3-9965-02c57f42164b" (UID: "ba3bd8c9-443a-45a3-9965-02c57f42164b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:08:01.477729 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.477702    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-podinfo" pod "ba3bd8c9-443a-45a3-9965-02c57f42164b" (UID: "ba3bd8c9-443a-45a3-9965-02c57f42164b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:08:01.477821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.477707    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-kube-api-access-rt79z" pod "ba3bd8c9-443a-45a3-9965-02c57f42164b" (UID: "ba3bd8c9-443a-45a3-9965-02c57f42164b"). InnerVolumeSpecName "kube-api-access-rt79z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:08:01.577250 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.577167    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.577250 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.577195    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.577250 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.577204    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.577250 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.577211    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3bd8c9-443a-45a3-9965-02c57f42164b-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.577250 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.577219    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3bd8c9-443a-45a3-9965-02c57f42164b-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.577250 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:01.577227    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt79z\" (UniqueName: \"kubernetes.io/projected/ba3bd8c9-443a-45a3-9965-02c57f42164b-kube-api-access-rt79z\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:02.050840 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:02.050807    5072 generic.go:356] "Generic (PLEG): container finished" podID="ba3bd8c9-443a-45a3-9965-02c57f42164b" containerID="c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4" exitCode=0
Oct 30 05:08:02.051028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:02.050851    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"
Oct 30 05:08:02.051028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:02.050891    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm" event={"ID":"ba3bd8c9-443a-45a3-9965-02c57f42164b","Type":"ContainerDied","Data":"c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4"}
Oct 30 05:08:02.051028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:02.050952    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm" event={"ID":"ba3bd8c9-443a-45a3-9965-02c57f42164b","Type":"ContainerDied","Data":"86bd6468cf73ca91ac2d6e9638014173f1fa562ac26b75317a24e8bc7967f7af"}
Oct 30 05:08:02.051028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:02.050981    5072 scope.go:122] "RemoveContainer" containerID="c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4"
Oct 30 05:08:02.063430 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:02.063415    5072 scope.go:122] "RemoveContainer" containerID="c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4"
Oct 30 05:08:02.063770 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:08:02.063749    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4\": container with ID starting with c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4 not found: ID does not exist" containerID="c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4"
Oct 30 05:08:02.063835 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:02.063779    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4"} err="failed to get container status \"c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4\": rpc error: code = NotFound desc = could not find container \"c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4\": container with ID starting with c0d5a76141756f0701ba662734352d8a9c5fe9f14f29fc381c54f4f7e2f626c4 not found: ID does not exist"
Oct 30 05:08:02.120547 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:02.120515    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"]
Oct 30 05:08:02.133097 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:02.133049    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-d2ksm"]
Oct 30 05:08:02.411626 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:02.411553    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3bd8c9-443a-45a3-9965-02c57f42164b" path="/var/lib/kubelet/pods/ba3bd8c9-443a-45a3-9965-02c57f42164b/volumes"
Oct 30 05:08:03.029258 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:03.029227    5072 scope.go:122] "RemoveContainer" containerID="7b45e055b8774f46ad46c12f0ecba1e6b867d7ebdaa2a544d1e9320cb0b78803"
Oct 30 05:08:03.043101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:03.043066    5072 scope.go:122] "RemoveContainer" containerID="49c727f733b38bb310472c23b22e402c0668f114e443b99407bf893eddbb7cf6"
Oct 30 05:08:03.063241 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:03.063207    5072 scope.go:122] "RemoveContainer" containerID="b667f8100d90d356c9f9cf8bc76c4d26fb4d69f5645e1aab912ff1d1c53b2842"
Oct 30 05:08:05.601496 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.601462    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"]
Oct 30 05:08:05.601894 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.601690    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8" podUID="7cd75cac-bcad-4d17-93fb-817d5576a32e" containerName="discovery" containerID="cri-o://6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528" gracePeriod=30
Oct 30 05:08:05.897286 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.897256    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 05:08:05.926850 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.926820    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-token\" (UniqueName: \"kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-token\") pod \"7cd75cac-bcad-4d17-93fb-817d5576a32e\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") "
Oct 30 05:08:05.927013 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.926857    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-kubeconfig\") pod \"7cd75cac-bcad-4d17-93fb-817d5576a32e\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") "
Oct 30 05:08:05.927013 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.926913    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-ca-configmap\") pod \"7cd75cac-bcad-4d17-93fb-817d5576a32e\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") "
Oct 30 05:08:05.927013 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.926982    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd75cac-bcad-4d17-93fb-817d5576a32e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd75cac-bcad-4d17-93fb-817d5576a32e-local-certs\") pod \"7cd75cac-bcad-4d17-93fb-817d5576a32e\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") "
Oct 30 05:08:05.927207 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.927020    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-dns-cert\") pod \"7cd75cac-bcad-4d17-93fb-817d5576a32e\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") "
Oct 30 05:08:05.927207 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.927134    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-kube-api-access-x7kbp\" (UniqueName: \"kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-kube-api-access-x7kbp\") pod \"7cd75cac-bcad-4d17-93fb-817d5576a32e\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") "
Oct 30 05:08:05.927330 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.927251    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-cacerts\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-cacerts\") pod \"7cd75cac-bcad-4d17-93fb-817d5576a32e\" (UID: \"7cd75cac-bcad-4d17-93fb-817d5576a32e\") "
Oct 30 05:08:05.927477 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.927451    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-ca-configmap" pod "7cd75cac-bcad-4d17-93fb-817d5576a32e" (UID: "7cd75cac-bcad-4d17-93fb-817d5576a32e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:08:05.927900 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.927860    5072 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-ca-configmap\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:05.929672 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.929640    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd75cac-bcad-4d17-93fb-817d5576a32e-local-certs" pod "7cd75cac-bcad-4d17-93fb-817d5576a32e" (UID: "7cd75cac-bcad-4d17-93fb-817d5576a32e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:08:05.929927 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.929902    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-dns-cert" pod "7cd75cac-bcad-4d17-93fb-817d5576a32e" (UID: "7cd75cac-bcad-4d17-93fb-817d5576a32e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:08:05.929998 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.929918    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-cacerts" pod "7cd75cac-bcad-4d17-93fb-817d5576a32e" (UID: "7cd75cac-bcad-4d17-93fb-817d5576a32e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:08:05.930060 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.930030    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-token" pod "7cd75cac-bcad-4d17-93fb-817d5576a32e" (UID: "7cd75cac-bcad-4d17-93fb-817d5576a32e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:08:05.930060 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.930041    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-kubeconfig" pod "7cd75cac-bcad-4d17-93fb-817d5576a32e" (UID: "7cd75cac-bcad-4d17-93fb-817d5576a32e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:08:05.930224 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:05.930153    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-kube-api-access-x7kbp" pod "7cd75cac-bcad-4d17-93fb-817d5576a32e" (UID: "7cd75cac-bcad-4d17-93fb-817d5576a32e"). InnerVolumeSpecName "kube-api-access-x7kbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:08:06.029023 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.028933    5072 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd75cac-bcad-4d17-93fb-817d5576a32e-local-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:06.029023 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.028966    5072 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-csr-dns-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:06.029023 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.028983    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7kbp\" (UniqueName: \"kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-kube-api-access-x7kbp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:06.029023 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.028996    5072 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-cacerts\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:06.029023 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.029008    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:06.029023 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.029021    5072 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cd75cac-bcad-4d17-93fb-817d5576a32e-istio-kubeconfig\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:06.097659 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.097620    5072 generic.go:356] "Generic (PLEG): container finished" podID="7cd75cac-bcad-4d17-93fb-817d5576a32e" containerID="6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528" exitCode=0
Oct 30 05:08:06.097857 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.097672    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"
Oct 30 05:08:06.097857 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.097701    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8" event={"ID":"7cd75cac-bcad-4d17-93fb-817d5576a32e","Type":"ContainerDied","Data":"6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528"}
Oct 30 05:08:06.097857 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.097760    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8" event={"ID":"7cd75cac-bcad-4d17-93fb-817d5576a32e","Type":"ContainerDied","Data":"e13e009a3644a38b5145bdd100fad38e064a235b3a1229bcb27916ccc616ab96"}
Oct 30 05:08:06.097857 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.097780    5072 scope.go:122] "RemoveContainer" containerID="6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528"
Oct 30 05:08:06.111190 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.111174    5072 scope.go:122] "RemoveContainer" containerID="6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528"
Oct 30 05:08:06.111427 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:08:06.111410    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528\": container with ID starting with 6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528 not found: ID does not exist" containerID="6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528"
Oct 30 05:08:06.111502 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.111434    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528"} err="failed to get container status \"6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528\": rpc error: code = NotFound desc = could not find container \"6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528\": container with ID starting with 6467d318d49c7ac4487a216653e83642fbf272a781dc23cef08ce1d845927528 not found: ID does not exist"
Oct 30 05:08:06.153147 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.153118    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"]
Oct 30 05:08:06.165308 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.165278    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9hb8"]
Oct 30 05:08:06.409636 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:06.409563    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd75cac-bcad-4d17-93fb-817d5576a32e" path="/var/lib/kubelet/pods/7cd75cac-bcad-4d17-93fb-817d5576a32e/volumes"
Oct 30 05:08:08.296085 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.296043    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"]
Oct 30 05:08:08.296751 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.296735    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ba3bd8c9-443a-45a3-9965-02c57f42164b" containerName="istio-proxy"
Oct 30 05:08:08.296804 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.296752    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3bd8c9-443a-45a3-9965-02c57f42164b" containerName="istio-proxy"
Oct 30 05:08:08.296804 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.296768    5072 cpu_manager.go:414] "containerMap: removing container" podUID="0674b215-1609-416c-8921-13b00c6ad543" containerName="revoke-keys"
Oct 30 05:08:08.296804 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.296774    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0674b215-1609-416c-8921-13b00c6ad543" containerName="revoke-keys"
Oct 30 05:08:08.296804 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.296800    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7cd75cac-bcad-4d17-93fb-817d5576a32e" containerName="discovery"
Oct 30 05:08:08.296926 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.296805    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd75cac-bcad-4d17-93fb-817d5576a32e" containerName="discovery"
Oct 30 05:08:08.296926 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.296923    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd75cac-bcad-4d17-93fb-817d5576a32e" containerName="discovery"
Oct 30 05:08:08.297000 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.296937    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3bd8c9-443a-45a3-9965-02c57f42164b" containerName="istio-proxy"
Oct 30 05:08:08.297000 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.296945    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0674b215-1609-416c-8921-13b00c6ad543" containerName="revoke-keys"
Oct 30 05:08:08.300578 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.300561    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.321039 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.321010    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"]
Oct 30 05:08:08.328195 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.328171    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b5496290-serving-cert\""
Oct 30 05:08:08.357429 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.357379    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-maas-api-tls\") pod \"maas-api-e2e-mt-b5496290-6d4b4f7447-564rk\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.357619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.357465    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-metrics-certs\") pod \"maas-api-e2e-mt-b5496290-6d4b4f7447-564rk\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.357619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.357542    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlm2g\" (UniqueName: \"kubernetes.io/projected/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-kube-api-access-rlm2g\") pod \"maas-api-e2e-mt-b5496290-6d4b4f7447-564rk\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.357826 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.357662    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-openshift-service-ca\") pod \"maas-api-e2e-mt-b5496290-6d4b4f7447-564rk\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.459083 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.459042    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-metrics-certs\") pod \"maas-api-e2e-mt-b5496290-6d4b4f7447-564rk\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.459269 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.459139    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlm2g\" (UniqueName: \"kubernetes.io/projected/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-kube-api-access-rlm2g\") pod \"maas-api-e2e-mt-b5496290-6d4b4f7447-564rk\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.459269 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.459216    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-openshift-service-ca\") pod \"maas-api-e2e-mt-b5496290-6d4b4f7447-564rk\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.459397 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.459317    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-maas-api-tls\") pod \"maas-api-e2e-mt-b5496290-6d4b4f7447-564rk\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.459937 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.459871    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-openshift-service-ca\") pod \"maas-api-e2e-mt-b5496290-6d4b4f7447-564rk\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.461764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.461740    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-maas-api-tls\") pod \"maas-api-e2e-mt-b5496290-6d4b4f7447-564rk\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.461918 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.461898    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-metrics-certs\") pod \"maas-api-e2e-mt-b5496290-6d4b4f7447-564rk\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.483383 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.483358    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlm2g\" (UniqueName: \"kubernetes.io/projected/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-kube-api-access-rlm2g\") pod \"maas-api-e2e-mt-b5496290-6d4b4f7447-564rk\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.617416 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.617325    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:08.937381 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:08:08.937345    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ea75c6_e6e6_4dc5_a1e7_2aa4c063d1f8.slice/crio-33c511e7f752b5005e58cd8940727248bc3c0965543da9bfc29df13290ef6937 WatchSource:0}: Error finding container 33c511e7f752b5005e58cd8940727248bc3c0965543da9bfc29df13290ef6937: Status 404 returned error can't find the container with id 33c511e7f752b5005e58cd8940727248bc3c0965543da9bfc29df13290ef6937
Oct 30 05:08:08.937652 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.937629    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"]
Oct 30 05:08:08.938729 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:08.938710    5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:08:09.124155 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:09.124117    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk" event={"ID":"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8","Type":"ContainerStarted","Data":"33c511e7f752b5005e58cd8940727248bc3c0965543da9bfc29df13290ef6937"}
Oct 30 05:08:10.132857 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:10.132821    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk" event={"ID":"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8","Type":"ContainerStarted","Data":"61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08"}
Oct 30 05:08:10.133280 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:10.132892    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:10.175582 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:10.175519    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk" podStartSLOduration=1.694907213 podStartE2EDuration="2.175503753s" totalImagesPullingTime="480.59654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:10.175229824 +0000 UTC m=+4269.916652923" watchObservedRunningTime="2026-10-30 05:08:10.175503753 +0000 UTC m=+4269.916926849"
Oct 30 05:08:11.753683 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:08:11.753641    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:08:16.146457 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:16.146427    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:08:19.618612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.618578    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"]
Oct 30 05:08:19.624399 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.624374    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.642934 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.642907    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"]
Oct 30 05:08:19.651160 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.651136    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-4c2415\"/\"kube-root-ca.crt\""
Oct 30 05:08:19.651319 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.651136    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-4c2415\"/\"openshift-service-ca.crt\""
Oct 30 05:08:19.653437 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.653419    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-4c2415\"/\"default-dockercfg-8gt4x\""
Oct 30 05:08:19.653437 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.653427    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-4c2415\"/\"rate-test-model-4c2415-kserve-self-signed-certs\""
Oct 30 05:08:19.709564 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.709521    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-dshm\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.709564 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.709556    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-model-cache\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.709844 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.709687    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98cp\" (UniqueName: \"kubernetes.io/projected/79eed7f5-2953-4479-9b27-0eefd7223518-kube-api-access-g98cp\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.709844 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.709819    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79eed7f5-2953-4479-9b27-0eefd7223518-tls-certs\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.709961 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.709860    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-home\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.710025 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.709961    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-tmp-dir\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.811435 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.811398    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-tmp-dir\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.811617 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.811505    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-dshm\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.811617 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.811533    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-model-cache\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.811721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.811621    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98cp\" (UniqueName: \"kubernetes.io/projected/79eed7f5-2953-4479-9b27-0eefd7223518-kube-api-access-g98cp\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.811776 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.811721    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79eed7f5-2953-4479-9b27-0eefd7223518-tls-certs\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.811840 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.811777    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-home\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.811903 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.811853    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-tmp-dir\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.812002 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.811980    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-model-cache\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.812355 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.812328    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-home\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.814574 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.814547    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-dshm\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.814719 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.814617    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79eed7f5-2953-4479-9b27-0eefd7223518-tls-certs\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.836067 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.836037    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98cp\" (UniqueName: \"kubernetes.io/projected/79eed7f5-2953-4479-9b27-0eefd7223518-kube-api-access-g98cp\") pod \"rate-test-model-4c2415-kserve-5b768b997c-hck8m\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") " pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:19.943305 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:19.943275    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:20.253679 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:20.253651    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"]
Oct 30 05:08:20.254842 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:08:20.254817    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79eed7f5_2953_4479_9b27_0eefd7223518.slice/crio-d59f08cd65ab5360cb924a3c0c2b9be53352534ebea52ba7cb389192fbf2e044 WatchSource:0}: Error finding container d59f08cd65ab5360cb924a3c0c2b9be53352534ebea52ba7cb389192fbf2e044: Status 404 returned error can't find the container with id d59f08cd65ab5360cb924a3c0c2b9be53352534ebea52ba7cb389192fbf2e044
Oct 30 05:08:21.246920 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:21.246878    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m" event={"ID":"79eed7f5-2953-4479-9b27-0eefd7223518","Type":"ContainerStarted","Data":"6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d"}
Oct 30 05:08:21.247374 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:21.246932    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m" event={"ID":"79eed7f5-2953-4479-9b27-0eefd7223518","Type":"ContainerStarted","Data":"d59f08cd65ab5360cb924a3c0c2b9be53352534ebea52ba7cb389192fbf2e044"}
Oct 30 05:08:21.292661 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:21.292612    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m" podStartSLOduration=2.2925972310000002 podStartE2EDuration="2.292597231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:21.2891671 +0000 UTC m=+4281.030590198" watchObservedRunningTime="2026-10-30 05:08:21.292597231 +0000 UTC m=+4281.034020324"
Oct 30 05:08:21.943512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:21.943480    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:25.030512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:25.030469    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"]
Oct 30 05:08:25.043013 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:25.042983    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-029657f7-r5vkn"]
Oct 30 05:08:26.408982 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:26.408941    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a18c910-bf54-4700-8581-4787b9e5372b" path="/var/lib/kubelet/pods/0a18c910-bf54-4700-8581-4787b9e5372b/volumes"
Oct 30 05:08:29.943583 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:29.943546    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:29.956968 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:29.956943    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:30.326510 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:30.326424    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:08:34.842747 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.842701    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"]
Oct 30 05:08:34.848648 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.848621    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:34.867971 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.867931    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"]
Oct 30 05:08:34.876661 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.876627    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-bbce72\"/\"kube-root-ca.crt\""
Oct 30 05:08:34.876935 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.876627    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-bbce72\"/\"openshift-service-ca.crt\""
Oct 30 05:08:34.877391 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.877366    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-bbce72\"/\"rate-test-model-bbce72-kserve-self-signed-certs\""
Oct 30 05:08:34.877640 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.877623    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-bbce72\"/\"default-dockercfg-hmp9d\""
Oct 30 05:08:34.898831 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.898788    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6sxw\" (UniqueName: \"kubernetes.io/projected/f046af00-47a2-4493-8b22-3b3bdc2b0d02-kube-api-access-h6sxw\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:34.899204 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.899167    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-home\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:34.899364 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.899234    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-model-cache\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:34.899364 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.899334    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-dshm\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:34.899614 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.899594    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tmp-dir\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:34.899700 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:34.899640    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tls-certs\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.000837 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.000792    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-dshm\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.001066 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.000887    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tmp-dir\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.001066 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.000925    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tls-certs\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.001239 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.001138    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6sxw\" (UniqueName: \"kubernetes.io/projected/f046af00-47a2-4493-8b22-3b3bdc2b0d02-kube-api-access-h6sxw\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.001239 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.001217    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-home\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.001360 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.001278    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-model-cache\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.001426 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.001369    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tmp-dir\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.001557 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.001527    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-home\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.001700 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.001649    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-model-cache\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.004255 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.004232    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-dshm\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.004439 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.004413    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tls-certs\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.027659 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.027464    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6sxw\" (UniqueName: \"kubernetes.io/projected/f046af00-47a2-4493-8b22-3b3bdc2b0d02-kube-api-access-h6sxw\") pod \"rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") " pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.164109 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.163990    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:35.480882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:35.480852    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"]
Oct 30 05:08:35.481909 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:08:35.481885    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf046af00_47a2_4493_8b22_3b3bdc2b0d02.slice/crio-a7e5ff9fb6797fb6a2ecef8e2c8430e83e0b57ad33217fa765180e9611d8ff77 WatchSource:0}: Error finding container a7e5ff9fb6797fb6a2ecef8e2c8430e83e0b57ad33217fa765180e9611d8ff77: Status 404 returned error can't find the container with id a7e5ff9fb6797fb6a2ecef8e2c8430e83e0b57ad33217fa765180e9611d8ff77
Oct 30 05:08:36.031168 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:36.031127    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"]
Oct 30 05:08:36.041371 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:36.041342    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbaef71c-b2rmk"]
Oct 30 05:08:36.410741 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:36.410664    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c25813-b497-4d0b-b77d-f9c0662b610d" path="/var/lib/kubelet/pods/79c25813-b497-4d0b-b77d-f9c0662b610d/volumes"
Oct 30 05:08:36.411017 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:36.410999    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk" event={"ID":"f046af00-47a2-4493-8b22-3b3bdc2b0d02","Type":"ContainerStarted","Data":"e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2"}
Oct 30 05:08:36.411061 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:36.411034    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk" event={"ID":"f046af00-47a2-4493-8b22-3b3bdc2b0d02","Type":"ContainerStarted","Data":"a7e5ff9fb6797fb6a2ecef8e2c8430e83e0b57ad33217fa765180e9611d8ff77"}
Oct 30 05:08:36.447021 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:36.446968    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk" podStartSLOduration=2.44695223 podStartE2EDuration="2.44695223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:36.445286911 +0000 UTC m=+4296.186710007" watchObservedRunningTime="2026-10-30 05:08:36.44695223 +0000 UTC m=+4296.188375328"
Oct 30 05:08:37.164303 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:37.164272    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:41.752845 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:08:41.752804    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:08:44.527155 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.527126    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"]
Oct 30 05:08:44.531724 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.531706    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.547672 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.547647    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-247158-openshift-default-dockercfg-rvw8n\""
Oct 30 05:08:44.564972 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.564944    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"]
Oct 30 05:08:44.620383 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.620345    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-envoy\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.620519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.620487    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-podinfo\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.620616 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.620598    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-certs\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.620680 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.620628    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-token\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.620763 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.620748    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.620826 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.620781    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-credential-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.620917 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.620896    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-data\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.621022 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.621005    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbq89\" (UniqueName: \"kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-kube-api-access-zbq89\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.621071 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.621058    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istiod-ca-cert\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.721806 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.721778    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istiod-ca-cert\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.721940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.721814    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-envoy\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.721940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.721862    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-podinfo\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.721940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.721903    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-certs\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.721940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.721917    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-token\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.722229 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.722041    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.722229 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.722110    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-credential-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.722229 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.722151    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-data\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.722409 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.722262    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbq89\" (UniqueName: \"kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-kube-api-access-zbq89\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.722409 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.722345    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-certs\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.722516 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.722492    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.722588 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.722564    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-data\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.722693 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.722673    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istiod-ca-cert\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.722744 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.722731    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-credential-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.724872 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.724850    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-envoy\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.725216 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.725194    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-podinfo\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.746362 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.746338    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-token\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.746621 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.746598    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbq89\" (UniqueName: \"kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-kube-api-access-zbq89\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:44.846855 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:44.846766    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:45.165003 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:45.164915    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:45.170693 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:45.170665    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"]
Oct 30 05:08:45.170958 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:08:45.170935    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbdae54_cfe9_4ced_a212_2cfc4e1372ce.slice/crio-6f637e95c503e2f977d14b64c0394d4cf114750b49521595d9b676ec3040e5af WatchSource:0}: Error finding container 6f637e95c503e2f977d14b64c0394d4cf114750b49521595d9b676ec3040e5af: Status 404 returned error can't find the container with id 6f637e95c503e2f977d14b64c0394d4cf114750b49521595d9b676ec3040e5af
Oct 30 05:08:45.173594 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:45.173565    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:08:45.173684 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:45.173626    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:08:45.173684 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:45.173656    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:08:45.179168 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:45.179146    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:45.476998 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:45.476959    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk" event={"ID":"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce","Type":"ContainerStarted","Data":"3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6"}
Oct 30 05:08:45.477247 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:45.477047    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk" event={"ID":"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce","Type":"ContainerStarted","Data":"6f637e95c503e2f977d14b64c0394d4cf114750b49521595d9b676ec3040e5af"}
Oct 30 05:08:45.488587 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:45.488562    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:08:45.524759 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:45.524711    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk" podStartSLOduration=1.524697974 podStartE2EDuration="1.524697974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:45.522716199 +0000 UTC m=+4305.264139306" watchObservedRunningTime="2026-10-30 05:08:45.524697974 +0000 UTC m=+4305.266121116"
Oct 30 05:08:45.847578 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:45.847499    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:46.254675 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:46.254644    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"]
Oct 30 05:08:46.287658 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:46.287620    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"]
Oct 30 05:08:46.848657 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:46.848620    5072 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 05:08:46.849052 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:46.848675    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk" podUID="4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 05:08:47.847695 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:47.847657    5072 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 05:08:47.848211 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:47.847739    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk" podUID="4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 05:08:48.847779 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:48.847745    5072 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 05:08:48.848254 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:48.847800    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk" podUID="4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 05:08:49.847924 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:49.847887    5072 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 05:08:49.848364 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:49.847953    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk" podUID="4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 05:08:50.847751 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:50.847714    5072 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 30 05:08:50.847917 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:50.847779    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk" podUID="4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 30 05:08:51.255430 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.255387    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55" podUID="a4f79ddb-940f-4136-bc52-9c4a8e99ac64" containerName="payload-processing" containerID="cri-o://fe88e0cfb8f0202f3252c125d2ddf6b6c8167a923b76b7f870ddf8fa2e54a94d" gracePeriod=25
Oct 30 05:08:51.288363 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.288292    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t" podUID="816b1bce-0e5a-4bd2-892a-6b128607ddef" containerName="payload-pre-processing" containerID="cri-o://369d24ddba59fbb3974e7e35b74f03c0175973d27fb915e692ac4a16b6d2d202" gracePeriod=25
Oct 30 05:08:51.594673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.594608    5072 generic.go:356] "Generic (PLEG): container finished" podID="816b1bce-0e5a-4bd2-892a-6b128607ddef" containerID="369d24ddba59fbb3974e7e35b74f03c0175973d27fb915e692ac4a16b6d2d202" exitCode=0
Oct 30 05:08:51.595234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.595177    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t" event={"ID":"816b1bce-0e5a-4bd2-892a-6b128607ddef","Type":"ContainerDied","Data":"369d24ddba59fbb3974e7e35b74f03c0175973d27fb915e692ac4a16b6d2d202"}
Oct 30 05:08:51.601764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.601732    5072 generic.go:356] "Generic (PLEG): container finished" podID="a4f79ddb-940f-4136-bc52-9c4a8e99ac64" containerID="fe88e0cfb8f0202f3252c125d2ddf6b6c8167a923b76b7f870ddf8fa2e54a94d" exitCode=0
Oct 30 05:08:51.602095 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.602053    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55" event={"ID":"a4f79ddb-940f-4136-bc52-9c4a8e99ac64","Type":"ContainerDied","Data":"fe88e0cfb8f0202f3252c125d2ddf6b6c8167a923b76b7f870ddf8fa2e54a94d"}
Oct 30 05:08:51.658055 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.658030    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"
Oct 30 05:08:51.661625 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.661602    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"
Oct 30 05:08:51.846879 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.846780    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816b1bce-0e5a-4bd2-892a-6b128607ddef-kube-api-access-2fclf\" (UniqueName: \"kubernetes.io/projected/816b1bce-0e5a-4bd2-892a-6b128607ddef-kube-api-access-2fclf\") pod \"816b1bce-0e5a-4bd2-892a-6b128607ddef\" (UID: \"816b1bce-0e5a-4bd2-892a-6b128607ddef\") "
Oct 30 05:08:51.846879 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.846845    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-plugins-config-volume\") pod \"a4f79ddb-940f-4136-bc52-9c4a8e99ac64\" (UID: \"a4f79ddb-940f-4136-bc52-9c4a8e99ac64\") "
Oct 30 05:08:51.847195 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.846911    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816b1bce-0e5a-4bd2-892a-6b128607ddef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816b1bce-0e5a-4bd2-892a-6b128607ddef-plugins-config-volume\") pod \"816b1bce-0e5a-4bd2-892a-6b128607ddef\" (UID: \"816b1bce-0e5a-4bd2-892a-6b128607ddef\") "
Oct 30 05:08:51.847195 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.846934    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-kube-api-access-v9lv5\" (UniqueName: \"kubernetes.io/projected/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-kube-api-access-v9lv5\") pod \"a4f79ddb-940f-4136-bc52-9c4a8e99ac64\" (UID: \"a4f79ddb-940f-4136-bc52-9c4a8e99ac64\") "
Oct 30 05:08:51.847322 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.847296    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-plugins-config-volume" pod "a4f79ddb-940f-4136-bc52-9c4a8e99ac64" (UID: "a4f79ddb-940f-4136-bc52-9c4a8e99ac64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:08:51.847389 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.847343    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816b1bce-0e5a-4bd2-892a-6b128607ddef-plugins-config-volume" pod "816b1bce-0e5a-4bd2-892a-6b128607ddef" (UID: "816b1bce-0e5a-4bd2-892a-6b128607ddef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:08:51.847506 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.847488    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816b1bce-0e5a-4bd2-892a-6b128607ddef-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:51.847588 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.847507    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:51.849407 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.849370    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816b1bce-0e5a-4bd2-892a-6b128607ddef-kube-api-access-2fclf" pod "816b1bce-0e5a-4bd2-892a-6b128607ddef" (UID: "816b1bce-0e5a-4bd2-892a-6b128607ddef"). InnerVolumeSpecName "kube-api-access-2fclf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:08:51.849521 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.849467    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-kube-api-access-v9lv5" pod "a4f79ddb-940f-4136-bc52-9c4a8e99ac64" (UID: "a4f79ddb-940f-4136-bc52-9c4a8e99ac64"). InnerVolumeSpecName "kube-api-access-v9lv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:08:51.852540 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.852517    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:51.852990 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.852973    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:51.853805 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.853782    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:08:51.948771 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.948732    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9lv5\" (UniqueName: \"kubernetes.io/projected/a4f79ddb-940f-4136-bc52-9c4a8e99ac64-kube-api-access-v9lv5\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:51.948771 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:51.948760    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fclf\" (UniqueName: \"kubernetes.io/projected/816b1bce-0e5a-4bd2-892a-6b128607ddef-kube-api-access-2fclf\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:08:52.613982 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:52.613899    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"
Oct 30 05:08:52.613982 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:52.613913    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t" event={"ID":"816b1bce-0e5a-4bd2-892a-6b128607ddef","Type":"ContainerDied","Data":"207018e491cac2355f770ced8e6a7abd176bded20dd3678a32bfd3188282cbea"}
Oct 30 05:08:52.614615 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:52.613990    5072 scope.go:122] "RemoveContainer" containerID="369d24ddba59fbb3974e7e35b74f03c0175973d27fb915e692ac4a16b6d2d202"
Oct 30 05:08:52.616521 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:52.616233    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55" event={"ID":"a4f79ddb-940f-4136-bc52-9c4a8e99ac64","Type":"ContainerDied","Data":"e3381395b8ee285e586f52e81f71938e2b6b5c0a80f293136a7432b16906ed4c"}
Oct 30 05:08:52.616521 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:52.616358    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"
Oct 30 05:08:52.626543 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:52.626521    5072 scope.go:122] "RemoveContainer" containerID="fe88e0cfb8f0202f3252c125d2ddf6b6c8167a923b76b7f870ddf8fa2e54a94d"
Oct 30 05:08:52.670053 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:52.670026    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"]
Oct 30 05:08:52.682851 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:52.682825    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ed8266-5d94cc9fcf-d6p8t"]
Oct 30 05:08:52.713464 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:52.713432    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"]
Oct 30 05:08:52.717995 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:52.717971    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ed8266-859f7dcc46-khc55"]
Oct 30 05:08:54.411532 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:54.411501    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816b1bce-0e5a-4bd2-892a-6b128607ddef" path="/var/lib/kubelet/pods/816b1bce-0e5a-4bd2-892a-6b128607ddef/volumes"
Oct 30 05:08:54.411964 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:08:54.411948    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f79ddb-940f-4136-bc52-9c4a8e99ac64" path="/var/lib/kubelet/pods/a4f79ddb-940f-4136-bc52-9c4a8e99ac64/volumes"
Oct 30 05:09:11.753048 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:09:11.753007    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:09:24.228302 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:24.228264    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"]
Oct 30 05:09:24.260449 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:24.260403    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"]
Oct 30 05:09:26.215764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:26.215723    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"]
Oct 30 05:09:26.227781 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:26.227747    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"]
Oct 30 05:09:29.229090 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:29.229047    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h" podUID="6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d" containerName="payload-processing" containerID="cri-o://ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36" gracePeriod=25
Oct 30 05:09:29.260728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:29.260673    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr" podUID="c3cb2b9f-1498-461d-bda4-4ddee9819453" containerName="payload-pre-processing" containerID="cri-o://88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06" gracePeriod=25
Oct 30 05:09:30.250506 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:30.250466    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr" podUID="c3cb2b9f-1498-461d-bda4-4ddee9819453" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:09:30.250506 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:30.250488    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h" podUID="6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:09:31.216452 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:31.216397    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q" podUID="5822e703-13ed-4d49-8a12-2f51750ccbe4" containerName="payload-processing" containerID="cri-o://13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5" gracePeriod=25
Oct 30 05:09:31.228131 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:31.228095    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh" podUID="514edbce-f4ec-41b9-9b7a-e6879381720d" containerName="payload-pre-processing" containerID="cri-o://4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b" gracePeriod=25
Oct 30 05:09:31.277698 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:31.277657    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q" podUID="5822e703-13ed-4d49-8a12-2f51750ccbe4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:09:31.277698 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:31.277674    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh" podUID="514edbce-f4ec-41b9-9b7a-e6879381720d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:09:40.250275 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:40.250192    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h" podUID="6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:09:40.250706 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:40.250198    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr" podUID="c3cb2b9f-1498-461d-bda4-4ddee9819453" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:09:41.277546 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:41.277502    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh" podUID="514edbce-f4ec-41b9-9b7a-e6879381720d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:09:41.277958 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:41.277522    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q" podUID="5822e703-13ed-4d49-8a12-2f51750ccbe4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:09:41.753554 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:09:41.753512    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:09:42.031028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:42.030939    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"]
Oct 30 05:09:42.042924 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:42.042891    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3907dfa9-pkslz"]
Oct 30 05:09:42.410258 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:42.410179    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="385b9725-5079-42e7-87b0-d361776db342" path="/var/lib/kubelet/pods/385b9725-5079-42e7-87b0-d361776db342/volumes"
Oct 30 05:09:49.549584 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.549554    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"
Oct 30 05:09:49.586702 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.586678    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"
Oct 30 05:09:49.624883 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.624850    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-kube-api-access-2f4gb\" (UniqueName: \"kubernetes.io/projected/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-kube-api-access-2f4gb\") pod \"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d\" (UID: \"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d\") "
Oct 30 05:09:49.625109 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.624960    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-plugins-config-volume\") pod \"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d\" (UID: \"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d\") "
Oct 30 05:09:49.625460 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.625431    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-plugins-config-volume" pod "6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d" (UID: "6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:09:49.625637 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.625619    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:09:49.627944 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.627921    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-kube-api-access-2f4gb" pod "6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d" (UID: "6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d"). InnerVolumeSpecName "kube-api-access-2f4gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:09:49.727118 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.727071    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3cb2b9f-1498-461d-bda4-4ddee9819453-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3cb2b9f-1498-461d-bda4-4ddee9819453-plugins-config-volume\") pod \"c3cb2b9f-1498-461d-bda4-4ddee9819453\" (UID: \"c3cb2b9f-1498-461d-bda4-4ddee9819453\") "
Oct 30 05:09:49.727325 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.727144    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cb2b9f-1498-461d-bda4-4ddee9819453-kube-api-access-m7c49\" (UniqueName: \"kubernetes.io/projected/c3cb2b9f-1498-461d-bda4-4ddee9819453-kube-api-access-m7c49\") pod \"c3cb2b9f-1498-461d-bda4-4ddee9819453\" (UID: \"c3cb2b9f-1498-461d-bda4-4ddee9819453\") "
Oct 30 05:09:49.727479 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.727465    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f4gb\" (UniqueName: \"kubernetes.io/projected/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d-kube-api-access-2f4gb\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:09:49.727554 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.727518    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cb2b9f-1498-461d-bda4-4ddee9819453-plugins-config-volume" pod "c3cb2b9f-1498-461d-bda4-4ddee9819453" (UID: "c3cb2b9f-1498-461d-bda4-4ddee9819453"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:09:49.729459 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.729440    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cb2b9f-1498-461d-bda4-4ddee9819453-kube-api-access-m7c49" pod "c3cb2b9f-1498-461d-bda4-4ddee9819453" (UID: "c3cb2b9f-1498-461d-bda4-4ddee9819453"). InnerVolumeSpecName "kube-api-access-m7c49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:09:49.829001 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.828924    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3cb2b9f-1498-461d-bda4-4ddee9819453-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:09:49.829001 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:49.828952    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7c49\" (UniqueName: \"kubernetes.io/projected/c3cb2b9f-1498-461d-bda4-4ddee9819453-kube-api-access-m7c49\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:09:50.053444 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.053407    5072 generic.go:356] "Generic (PLEG): container finished" podID="c3cb2b9f-1498-461d-bda4-4ddee9819453" containerID="88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06" exitCode=0
Oct 30 05:09:50.053444 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.053459    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"
Oct 30 05:09:50.053679 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.053493    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr" event={"ID":"c3cb2b9f-1498-461d-bda4-4ddee9819453","Type":"ContainerDied","Data":"88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06"}
Oct 30 05:09:50.053679 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.053550    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr" event={"ID":"c3cb2b9f-1498-461d-bda4-4ddee9819453","Type":"ContainerDied","Data":"832fa879dfb6cfa2ad33e176cb6b623620dfa03757d611be6f0d3b4474c0b59d"}
Oct 30 05:09:50.053679 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.053574    5072 scope.go:122] "RemoveContainer" containerID="88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06"
Oct 30 05:09:50.055112 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.055057    5072 generic.go:356] "Generic (PLEG): container finished" podID="6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d" containerID="ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36" exitCode=0
Oct 30 05:09:50.055266 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.055111    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h" event={"ID":"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d","Type":"ContainerDied","Data":"ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36"}
Oct 30 05:09:50.055266 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.055131    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"
Oct 30 05:09:50.055266 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.055160    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h" event={"ID":"6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d","Type":"ContainerDied","Data":"90d6789d28d3afeec63a1a558631d6f168042586b9718f8fe94b2b2558fd9124"}
Oct 30 05:09:50.066501 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.066486    5072 scope.go:122] "RemoveContainer" containerID="88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06"
Oct 30 05:09:50.066782 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:09:50.066761    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06\": container with ID starting with 88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06 not found: ID does not exist" containerID="88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06"
Oct 30 05:09:50.066937 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.066789    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06"} err="failed to get container status \"88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06\": rpc error: code = NotFound desc = could not find container \"88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06\": container with ID starting with 88e8a37aed1fc0e21f14cba29e78b569d3a12dd3f3bca08ef948becd3e8a1f06 not found: ID does not exist"
Oct 30 05:09:50.066937 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.066808    5072 scope.go:122] "RemoveContainer" containerID="ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36"
Oct 30 05:09:50.077627 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.077594    5072 scope.go:122] "RemoveContainer" containerID="ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36"
Oct 30 05:09:50.078006 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:09:50.077966    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36\": container with ID starting with ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36 not found: ID does not exist" containerID="ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36"
Oct 30 05:09:50.078132 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.078017    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36"} err="failed to get container status \"ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36\": rpc error: code = NotFound desc = could not find container \"ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36\": container with ID starting with ad4135e23066740e71f5569c03bd736c5775baa65a9208d29c8a27c1c11a7c36 not found: ID does not exist"
Oct 30 05:09:50.108062 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.107972    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"]
Oct 30 05:09:50.114068 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.114034    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6378cc-69488d7976-q7h5h"]
Oct 30 05:09:50.149797 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.149758    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"]
Oct 30 05:09:50.153661 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.153625    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6378cc-75c547648c-m7rxr"]
Oct 30 05:09:50.409274 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.409202    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d" path="/var/lib/kubelet/pods/6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d/volumes"
Oct 30 05:09:50.409640 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:50.409626    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3cb2b9f-1498-461d-bda4-4ddee9819453" path="/var/lib/kubelet/pods/c3cb2b9f-1498-461d-bda4-4ddee9819453/volumes"
Oct 30 05:09:51.276299 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:09:51.276268    5072 logging.go:55] [core] [Channel #13086 SubChannel #13087]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.135:50052", ServerName: "10.132.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.135:50052: connect: connection refused"
Oct 30 05:09:51.276686 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:09:51.276382    5072 logging.go:55] [core] [Channel #13088 SubChannel #13089]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.134:50052", ServerName: "10.132.0.134:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.134:50052: connect: connection refused"
Oct 30 05:09:51.509509 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.509478    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"
Oct 30 05:09:51.555104 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.555039    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"
Oct 30 05:09:51.652216 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.652178    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5822e703-13ed-4d49-8a12-2f51750ccbe4-kube-api-access-6lnwc\" (UniqueName: \"kubernetes.io/projected/5822e703-13ed-4d49-8a12-2f51750ccbe4-kube-api-access-6lnwc\") pod \"5822e703-13ed-4d49-8a12-2f51750ccbe4\" (UID: \"5822e703-13ed-4d49-8a12-2f51750ccbe4\") "
Oct 30 05:09:51.652216 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.652216    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5822e703-13ed-4d49-8a12-2f51750ccbe4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5822e703-13ed-4d49-8a12-2f51750ccbe4-plugins-config-volume\") pod \"5822e703-13ed-4d49-8a12-2f51750ccbe4\" (UID: \"5822e703-13ed-4d49-8a12-2f51750ccbe4\") "
Oct 30 05:09:51.652492 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.652242    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514edbce-f4ec-41b9-9b7a-e6879381720d-kube-api-access-hwgbh\" (UniqueName: \"kubernetes.io/projected/514edbce-f4ec-41b9-9b7a-e6879381720d-kube-api-access-hwgbh\") pod \"514edbce-f4ec-41b9-9b7a-e6879381720d\" (UID: \"514edbce-f4ec-41b9-9b7a-e6879381720d\") "
Oct 30 05:09:51.652563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.652489    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514edbce-f4ec-41b9-9b7a-e6879381720d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514edbce-f4ec-41b9-9b7a-e6879381720d-plugins-config-volume\") pod \"514edbce-f4ec-41b9-9b7a-e6879381720d\" (UID: \"514edbce-f4ec-41b9-9b7a-e6879381720d\") "
Oct 30 05:09:51.652645 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.652605    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5822e703-13ed-4d49-8a12-2f51750ccbe4-plugins-config-volume" pod "5822e703-13ed-4d49-8a12-2f51750ccbe4" (UID: "5822e703-13ed-4d49-8a12-2f51750ccbe4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:09:51.652875 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.652846    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514edbce-f4ec-41b9-9b7a-e6879381720d-plugins-config-volume" pod "514edbce-f4ec-41b9-9b7a-e6879381720d" (UID: "514edbce-f4ec-41b9-9b7a-e6879381720d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:09:51.653238 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.653217    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514edbce-f4ec-41b9-9b7a-e6879381720d-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:09:51.653294 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.653245    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5822e703-13ed-4d49-8a12-2f51750ccbe4-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:09:51.654630 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.654612    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5822e703-13ed-4d49-8a12-2f51750ccbe4-kube-api-access-6lnwc" pod "5822e703-13ed-4d49-8a12-2f51750ccbe4" (UID: "5822e703-13ed-4d49-8a12-2f51750ccbe4"). InnerVolumeSpecName "kube-api-access-6lnwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:09:51.654688 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.654660    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514edbce-f4ec-41b9-9b7a-e6879381720d-kube-api-access-hwgbh" pod "514edbce-f4ec-41b9-9b7a-e6879381720d" (UID: "514edbce-f4ec-41b9-9b7a-e6879381720d"). InnerVolumeSpecName "kube-api-access-hwgbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:09:51.755009 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.754923    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lnwc\" (UniqueName: \"kubernetes.io/projected/5822e703-13ed-4d49-8a12-2f51750ccbe4-kube-api-access-6lnwc\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:09:51.755009 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:51.754963    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwgbh\" (UniqueName: \"kubernetes.io/projected/514edbce-f4ec-41b9-9b7a-e6879381720d-kube-api-access-hwgbh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:09:52.073431 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.073346    5072 generic.go:356] "Generic (PLEG): container finished" podID="5822e703-13ed-4d49-8a12-2f51750ccbe4" containerID="13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5" exitCode=0
Oct 30 05:09:52.073431 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.073409    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"
Oct 30 05:09:52.073674 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.073430    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q" event={"ID":"5822e703-13ed-4d49-8a12-2f51750ccbe4","Type":"ContainerDied","Data":"13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5"}
Oct 30 05:09:52.073674 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.073518    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q" event={"ID":"5822e703-13ed-4d49-8a12-2f51750ccbe4","Type":"ContainerDied","Data":"862639bc0729e0b1b7a3a3fd8a532b4f7bd8b6f040acb9d6d3f1989a35342180"}
Oct 30 05:09:52.073674 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.073546    5072 scope.go:122] "RemoveContainer" containerID="13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5"
Oct 30 05:09:52.074990 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.074964    5072 generic.go:356] "Generic (PLEG): container finished" podID="514edbce-f4ec-41b9-9b7a-e6879381720d" containerID="4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b" exitCode=0
Oct 30 05:09:52.075193 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.074993    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh" event={"ID":"514edbce-f4ec-41b9-9b7a-e6879381720d","Type":"ContainerDied","Data":"4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b"}
Oct 30 05:09:52.075193 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.075020    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"
Oct 30 05:09:52.075193 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.075042    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh" event={"ID":"514edbce-f4ec-41b9-9b7a-e6879381720d","Type":"ContainerDied","Data":"a6dbacabbfc445b8625ea9a5609b3f38472e6b374a5b841acfc870d065a4c1c2"}
Oct 30 05:09:52.089869 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.089852    5072 scope.go:122] "RemoveContainer" containerID="13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5"
Oct 30 05:09:52.090187 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:09:52.090169    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5\": container with ID starting with 13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5 not found: ID does not exist" containerID="13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5"
Oct 30 05:09:52.090259 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.090196    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5"} err="failed to get container status \"13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5\": rpc error: code = NotFound desc = could not find container \"13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5\": container with ID starting with 13221078757d0b0c8628ba2059c9aec9e20723bf984393ef143860e2d9cdd1e5 not found: ID does not exist"
Oct 30 05:09:52.090259 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.090215    5072 scope.go:122] "RemoveContainer" containerID="4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b"
Oct 30 05:09:52.099989 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.099971    5072 scope.go:122] "RemoveContainer" containerID="4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b"
Oct 30 05:09:52.100281 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:09:52.100261    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b\": container with ID starting with 4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b not found: ID does not exist" containerID="4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b"
Oct 30 05:09:52.100356 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.100287    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b"} err="failed to get container status \"4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b\": rpc error: code = NotFound desc = could not find container \"4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b\": container with ID starting with 4b43206004081c399560c52a404495dc95ce9aeecde76adb0ff7b0a33be6b63b not found: ID does not exist"
Oct 30 05:09:52.142786 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.142762    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"]
Oct 30 05:09:52.146561 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.146527    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q"]
Oct 30 05:09:52.183563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.183537    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"]
Oct 30 05:09:52.190499 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.190468    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh"]
Oct 30 05:09:52.276939 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.276895    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-483bea-64db95b77f-mprvh" podUID="514edbce-f4ec-41b9-9b7a-e6879381720d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.134:50052\" within 1s: context deadline exceeded"
Oct 30 05:09:52.277354 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.277046    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-483bea-585b8c58dd-zzv7q" podUID="5822e703-13ed-4d49-8a12-2f51750ccbe4" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.135:50052\" within 1s: context deadline exceeded"
Oct 30 05:09:52.277354 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:09:52.277160    5072 logging.go:55] [core] [Channel #13086 SubChannel #13087]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.135:50052", ServerName: "10.132.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.135:50052: operation was canceled"
Oct 30 05:09:52.408983 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.408910    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514edbce-f4ec-41b9-9b7a-e6879381720d" path="/var/lib/kubelet/pods/514edbce-f4ec-41b9-9b7a-e6879381720d/volumes"
Oct 30 05:09:52.409481 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:09:52.409459    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5822e703-13ed-4d49-8a12-2f51750ccbe4" path="/var/lib/kubelet/pods/5822e703-13ed-4d49-8a12-2f51750ccbe4/volumes"
Oct 30 05:10:03.137013 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:03.136980    5072 scope.go:122] "RemoveContainer" containerID="d98ec538ac69ef288d2d297cde94430e5468758ec4e27cb804c17e1ef44cdb11"
Oct 30 05:10:03.147606 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:03.147582    5072 scope.go:122] "RemoveContainer" containerID="cc16255f3fca9d7f5ee322091c7a850427d2e81cdf86971dae689131e65ffa6d"
Oct 30 05:10:10.699829 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.699792    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"]
Oct 30 05:10:10.700525 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700508    5072 cpu_manager.go:414] "containerMap: removing container" podUID="514edbce-f4ec-41b9-9b7a-e6879381720d" containerName="payload-pre-processing"
Oct 30 05:10:10.700581 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700525    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514edbce-f4ec-41b9-9b7a-e6879381720d" containerName="payload-pre-processing"
Oct 30 05:10:10.700581 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700543    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a4f79ddb-940f-4136-bc52-9c4a8e99ac64" containerName="payload-processing"
Oct 30 05:10:10.700581 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700548    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f79ddb-940f-4136-bc52-9c4a8e99ac64" containerName="payload-processing"
Oct 30 05:10:10.700581 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700565    5072 cpu_manager.go:414] "containerMap: removing container" podUID="816b1bce-0e5a-4bd2-892a-6b128607ddef" containerName="payload-pre-processing"
Oct 30 05:10:10.700581 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700569    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816b1bce-0e5a-4bd2-892a-6b128607ddef" containerName="payload-pre-processing"
Oct 30 05:10:10.700581 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700579    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5822e703-13ed-4d49-8a12-2f51750ccbe4" containerName="payload-processing"
Oct 30 05:10:10.700770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700585    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5822e703-13ed-4d49-8a12-2f51750ccbe4" containerName="payload-processing"
Oct 30 05:10:10.700770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700596    5072 cpu_manager.go:414] "containerMap: removing container" podUID="6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d" containerName="payload-processing"
Oct 30 05:10:10.700770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700600    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d" containerName="payload-processing"
Oct 30 05:10:10.700770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700610    5072 cpu_manager.go:414] "containerMap: removing container" podUID="c3cb2b9f-1498-461d-bda4-4ddee9819453" containerName="payload-pre-processing"
Oct 30 05:10:10.700770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700614    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cb2b9f-1498-461d-bda4-4ddee9819453" containerName="payload-pre-processing"
Oct 30 05:10:10.700770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700711    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="816b1bce-0e5a-4bd2-892a-6b128607ddef" containerName="payload-pre-processing"
Oct 30 05:10:10.700770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700719    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5822e703-13ed-4d49-8a12-2f51750ccbe4" containerName="payload-processing"
Oct 30 05:10:10.700770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700726    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae8ce0e-895c-4c4f-826d-ddffa5a16e4d" containerName="payload-processing"
Oct 30 05:10:10.700770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700733    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cb2b9f-1498-461d-bda4-4ddee9819453" containerName="payload-pre-processing"
Oct 30 05:10:10.700770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700744    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="514edbce-f4ec-41b9-9b7a-e6879381720d" containerName="payload-pre-processing"
Oct 30 05:10:10.700770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.700750    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f79ddb-940f-4136-bc52-9c4a8e99ac64" containerName="payload-processing"
Oct 30 05:10:10.704337 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.704320    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:10.725389 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.725362    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"]
Oct 30 05:10:10.735305 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.735280    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:10:10.735450 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.735397    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:10:10.797643 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.797616    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f010de4b-hjr5z\" (UID: \"0bc76efb-e407-4e83-b4cb-ca9a947def75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:10.797806 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.797757    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f010de4b-hjr5z\" (UID: \"0bc76efb-e407-4e83-b4cb-ca9a947def75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:10.797907 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.797886    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhd2x\" (UniqueName: \"kubernetes.io/projected/0bc76efb-e407-4e83-b4cb-ca9a947def75-kube-api-access-jhd2x\") pod \"maas-api-revoke-sub-f010de4b-hjr5z\" (UID: \"0bc76efb-e407-4e83-b4cb-ca9a947def75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:10.898983 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.898945    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f010de4b-hjr5z\" (UID: \"0bc76efb-e407-4e83-b4cb-ca9a947def75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:10.899206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.899047    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f010de4b-hjr5z\" (UID: \"0bc76efb-e407-4e83-b4cb-ca9a947def75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:10.899206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.899160    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhd2x\" (UniqueName: \"kubernetes.io/projected/0bc76efb-e407-4e83-b4cb-ca9a947def75-kube-api-access-jhd2x\") pod \"maas-api-revoke-sub-f010de4b-hjr5z\" (UID: \"0bc76efb-e407-4e83-b4cb-ca9a947def75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:10.899753 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.899727    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f010de4b-hjr5z\" (UID: \"0bc76efb-e407-4e83-b4cb-ca9a947def75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:10.901485 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.901463    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f010de4b-hjr5z\" (UID: \"0bc76efb-e407-4e83-b4cb-ca9a947def75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:10.927261 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:10.927230    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhd2x\" (UniqueName: \"kubernetes.io/projected/0bc76efb-e407-4e83-b4cb-ca9a947def75-kube-api-access-jhd2x\") pod \"maas-api-revoke-sub-f010de4b-hjr5z\" (UID: \"0bc76efb-e407-4e83-b4cb-ca9a947def75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:11.018834 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:11.018756    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:11.339359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:11.339325    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"]
Oct 30 05:10:11.339624 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:10:11.339603    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc76efb_e407_4e83_b4cb_ca9a947def75.slice/crio-e4de8bb763f2abf14464da3de3046d568ca7525838c804620445f71eeedcf02d WatchSource:0}: Error finding container e4de8bb763f2abf14464da3de3046d568ca7525838c804620445f71eeedcf02d: Status 404 returned error can't find the container with id e4de8bb763f2abf14464da3de3046d568ca7525838c804620445f71eeedcf02d
Oct 30 05:10:11.752539 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:10:11.752504    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:10:12.250769 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:12.250716    5072 generic.go:356] "Generic (PLEG): container finished" podID="0bc76efb-e407-4e83-b4cb-ca9a947def75" containerID="f9875cc1bab19b3fe3878871d90cb040239aaa9028658ed4efbafaf5a781ab81" exitCode=0
Oct 30 05:10:12.250769 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:12.250754    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z" event={"ID":"0bc76efb-e407-4e83-b4cb-ca9a947def75","Type":"ContainerDied","Data":"f9875cc1bab19b3fe3878871d90cb040239aaa9028658ed4efbafaf5a781ab81"}
Oct 30 05:10:12.250989 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:12.250815    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z" event={"ID":"0bc76efb-e407-4e83-b4cb-ca9a947def75","Type":"ContainerStarted","Data":"e4de8bb763f2abf14464da3de3046d568ca7525838c804620445f71eeedcf02d"}
Oct 30 05:10:13.421995 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:13.421959    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:13.532776 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:13.532698    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc76efb-e407-4e83-b4cb-ca9a947def75-kube-api-access-jhd2x\" (UniqueName: \"kubernetes.io/projected/0bc76efb-e407-4e83-b4cb-ca9a947def75-kube-api-access-jhd2x\") pod \"0bc76efb-e407-4e83-b4cb-ca9a947def75\" (UID: \"0bc76efb-e407-4e83-b4cb-ca9a947def75\") "
Oct 30 05:10:13.532925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:13.532813    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-xks-ca\") pod \"0bc76efb-e407-4e83-b4cb-ca9a947def75\" (UID: \"0bc76efb-e407-4e83-b4cb-ca9a947def75\") "
Oct 30 05:10:13.533003 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:13.532990    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-openshift-ca\") pod \"0bc76efb-e407-4e83-b4cb-ca9a947def75\" (UID: \"0bc76efb-e407-4e83-b4cb-ca9a947def75\") "
Oct 30 05:10:13.533453 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:13.533356    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-openshift-ca" pod "0bc76efb-e407-4e83-b4cb-ca9a947def75" (UID: "0bc76efb-e407-4e83-b4cb-ca9a947def75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:13.533680 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:13.533666    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:13.535024 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:13.535000    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-xks-ca" pod "0bc76efb-e407-4e83-b4cb-ca9a947def75" (UID: "0bc76efb-e407-4e83-b4cb-ca9a947def75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:13.535164 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:13.535119    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc76efb-e407-4e83-b4cb-ca9a947def75-kube-api-access-jhd2x" pod "0bc76efb-e407-4e83-b4cb-ca9a947def75" (UID: "0bc76efb-e407-4e83-b4cb-ca9a947def75"). InnerVolumeSpecName "kube-api-access-jhd2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:13.634886 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:13.634847    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc76efb-e407-4e83-b4cb-ca9a947def75-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:13.634886 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:13.634883    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhd2x\" (UniqueName: \"kubernetes.io/projected/0bc76efb-e407-4e83-b4cb-ca9a947def75-kube-api-access-jhd2x\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:14.265455 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:14.265425    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"
Oct 30 05:10:14.265455 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:14.265438    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z" event={"ID":"0bc76efb-e407-4e83-b4cb-ca9a947def75","Type":"ContainerDied","Data":"e4de8bb763f2abf14464da3de3046d568ca7525838c804620445f71eeedcf02d"}
Oct 30 05:10:14.265711 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:14.265504    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4de8bb763f2abf14464da3de3046d568ca7525838c804620445f71eeedcf02d"
Oct 30 05:10:14.633883 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:14.633787    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"]
Oct 30 05:10:14.670508 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:14.670474    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"]
Oct 30 05:10:19.634670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:19.634625    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2" podUID="ee7d87cd-c93b-49d6-8319-af928d231319" containerName="payload-processing" containerID="cri-o://4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d" gracePeriod=25
Oct 30 05:10:19.671570 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:19.671495    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx" podUID="620fa980-d930-44fe-b306-5db27ed9e675" containerName="payload-pre-processing" containerID="cri-o://00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a" gracePeriod=25
Oct 30 05:10:21.122637 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.122601    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"]
Oct 30 05:10:21.123328 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.123313    5072 cpu_manager.go:414] "containerMap: removing container" podUID="0bc76efb-e407-4e83-b4cb-ca9a947def75" containerName="revoke-keys"
Oct 30 05:10:21.123388 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.123329    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc76efb-e407-4e83-b4cb-ca9a947def75" containerName="revoke-keys"
Oct 30 05:10:21.123472 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.123462    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc76efb-e407-4e83-b4cb-ca9a947def75" containerName="revoke-keys"
Oct 30 05:10:21.127331 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.127311    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:21.150825 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.150799    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"]
Oct 30 05:10:21.157272 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.157246    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:10:21.157436 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.157417    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:10:21.232628 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.232592    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87350580-849d-40cd-a8f6-618c56f16493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60f00c59-w9phh\" (UID: \"87350580-849d-40cd-a8f6-618c56f16493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:21.232821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.232649    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzrvt\" (UniqueName: \"kubernetes.io/projected/87350580-849d-40cd-a8f6-618c56f16493-kube-api-access-jzrvt\") pod \"maas-api-revoke-sub-60f00c59-w9phh\" (UID: \"87350580-849d-40cd-a8f6-618c56f16493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:21.232821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.232779    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87350580-849d-40cd-a8f6-618c56f16493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60f00c59-w9phh\" (UID: \"87350580-849d-40cd-a8f6-618c56f16493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:21.333778 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.333744    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87350580-849d-40cd-a8f6-618c56f16493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60f00c59-w9phh\" (UID: \"87350580-849d-40cd-a8f6-618c56f16493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:21.333990 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.333813    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrvt\" (UniqueName: \"kubernetes.io/projected/87350580-849d-40cd-a8f6-618c56f16493-kube-api-access-jzrvt\") pod \"maas-api-revoke-sub-60f00c59-w9phh\" (UID: \"87350580-849d-40cd-a8f6-618c56f16493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:21.333990 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.333911    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87350580-849d-40cd-a8f6-618c56f16493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60f00c59-w9phh\" (UID: \"87350580-849d-40cd-a8f6-618c56f16493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:21.334635 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.334609    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87350580-849d-40cd-a8f6-618c56f16493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60f00c59-w9phh\" (UID: \"87350580-849d-40cd-a8f6-618c56f16493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:21.336380 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.336360    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87350580-849d-40cd-a8f6-618c56f16493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60f00c59-w9phh\" (UID: \"87350580-849d-40cd-a8f6-618c56f16493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:21.358043 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.358016    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrvt\" (UniqueName: \"kubernetes.io/projected/87350580-849d-40cd-a8f6-618c56f16493-kube-api-access-jzrvt\") pod \"maas-api-revoke-sub-60f00c59-w9phh\" (UID: \"87350580-849d-40cd-a8f6-618c56f16493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:21.442442 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.442411    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:21.761032 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:21.761005    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"]
Oct 30 05:10:21.762048 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:10:21.762024    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87350580_849d_40cd_a8f6_618c56f16493.slice/crio-1c5aff7f34b71a5244e9d8f56bfbb453e298d4d62d4bfbcfe9ee5d0201a17f0a WatchSource:0}: Error finding container 1c5aff7f34b71a5244e9d8f56bfbb453e298d4d62d4bfbcfe9ee5d0201a17f0a: Status 404 returned error can't find the container with id 1c5aff7f34b71a5244e9d8f56bfbb453e298d4d62d4bfbcfe9ee5d0201a17f0a
Oct 30 05:10:22.020066 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:22.019980    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx" podUID="620fa980-d930-44fe-b306-5db27ed9e675" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:10:22.020243 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:22.019992    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2" podUID="ee7d87cd-c93b-49d6-8319-af928d231319" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:10:22.326339 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:22.326247    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh" event={"ID":"87350580-849d-40cd-a8f6-618c56f16493","Type":"ContainerStarted","Data":"d1c7c537e2e1b1c9b113e5b5f61299c183a1ee7d33b9b5ebe9a5fe34c83d8933"}
Oct 30 05:10:22.326339 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:22.326311    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh" event={"ID":"87350580-849d-40cd-a8f6-618c56f16493","Type":"ContainerStarted","Data":"1c5aff7f34b71a5244e9d8f56bfbb453e298d4d62d4bfbcfe9ee5d0201a17f0a"}
Oct 30 05:10:22.369028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:22.368976    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh" podStartSLOduration=1.36896182 podStartE2EDuration="1.36896182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:22.368683704 +0000 UTC m=+4402.110106801" watchObservedRunningTime="2026-10-30 05:10:22.36896182 +0000 UTC m=+4402.110384916"
Oct 30 05:10:23.337708 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:23.337672    5072 generic.go:356] "Generic (PLEG): container finished" podID="87350580-849d-40cd-a8f6-618c56f16493" containerID="d1c7c537e2e1b1c9b113e5b5f61299c183a1ee7d33b9b5ebe9a5fe34c83d8933" exitCode=0
Oct 30 05:10:23.338391 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:23.337741    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh" event={"ID":"87350580-849d-40cd-a8f6-618c56f16493","Type":"ContainerDied","Data":"d1c7c537e2e1b1c9b113e5b5f61299c183a1ee7d33b9b5ebe9a5fe34c83d8933"}
Oct 30 05:10:24.498001 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:24.497977    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:24.584143 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:24.584103    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87350580-849d-40cd-a8f6-618c56f16493-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87350580-849d-40cd-a8f6-618c56f16493-maas-api-xks-ca\") pod \"87350580-849d-40cd-a8f6-618c56f16493\" (UID: \"87350580-849d-40cd-a8f6-618c56f16493\") "
Oct 30 05:10:24.584332 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:24.584189    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87350580-849d-40cd-a8f6-618c56f16493-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87350580-849d-40cd-a8f6-618c56f16493-maas-api-openshift-ca\") pod \"87350580-849d-40cd-a8f6-618c56f16493\" (UID: \"87350580-849d-40cd-a8f6-618c56f16493\") "
Oct 30 05:10:24.584332 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:24.584259    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87350580-849d-40cd-a8f6-618c56f16493-kube-api-access-jzrvt\" (UniqueName: \"kubernetes.io/projected/87350580-849d-40cd-a8f6-618c56f16493-kube-api-access-jzrvt\") pod \"87350580-849d-40cd-a8f6-618c56f16493\" (UID: \"87350580-849d-40cd-a8f6-618c56f16493\") "
Oct 30 05:10:24.584612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:24.584581    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87350580-849d-40cd-a8f6-618c56f16493-maas-api-openshift-ca" pod "87350580-849d-40cd-a8f6-618c56f16493" (UID: "87350580-849d-40cd-a8f6-618c56f16493"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:24.584907 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:24.584861    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87350580-849d-40cd-a8f6-618c56f16493-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:24.586524 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:24.586495    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87350580-849d-40cd-a8f6-618c56f16493-maas-api-xks-ca" pod "87350580-849d-40cd-a8f6-618c56f16493" (UID: "87350580-849d-40cd-a8f6-618c56f16493"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:24.586970 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:24.586948    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87350580-849d-40cd-a8f6-618c56f16493-kube-api-access-jzrvt" pod "87350580-849d-40cd-a8f6-618c56f16493" (UID: "87350580-849d-40cd-a8f6-618c56f16493"). InnerVolumeSpecName "kube-api-access-jzrvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:24.686194 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:24.686166    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87350580-849d-40cd-a8f6-618c56f16493-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:24.686194 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:24.686192    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzrvt\" (UniqueName: \"kubernetes.io/projected/87350580-849d-40cd-a8f6-618c56f16493-kube-api-access-jzrvt\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:25.030940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:25.030908    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"]
Oct 30 05:10:25.043557 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:25.043525    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1122785c-7n75f"]
Oct 30 05:10:25.352569 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:25.352492    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"
Oct 30 05:10:25.352569 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:25.352517    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh" event={"ID":"87350580-849d-40cd-a8f6-618c56f16493","Type":"ContainerDied","Data":"1c5aff7f34b71a5244e9d8f56bfbb453e298d4d62d4bfbcfe9ee5d0201a17f0a"}
Oct 30 05:10:25.352755 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:25.352594    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c5aff7f34b71a5244e9d8f56bfbb453e298d4d62d4bfbcfe9ee5d0201a17f0a"
Oct 30 05:10:26.409070 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:26.409038    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8408b0f4-ac8e-4d4f-afaf-b67f547dcc41" path="/var/lib/kubelet/pods/8408b0f4-ac8e-4d4f-afaf-b67f547dcc41/volumes"
Oct 30 05:10:32.020370 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.020326    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2" podUID="ee7d87cd-c93b-49d6-8319-af928d231319" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:10:32.020859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.020373    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx" podUID="620fa980-d930-44fe-b306-5db27ed9e675" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:10:32.024141 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.024120    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"]
Oct 30 05:10:32.024806 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.024792    5072 cpu_manager.go:414] "containerMap: removing container" podUID="87350580-849d-40cd-a8f6-618c56f16493" containerName="revoke-keys"
Oct 30 05:10:32.024852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.024807    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87350580-849d-40cd-a8f6-618c56f16493" containerName="revoke-keys"
Oct 30 05:10:32.024942 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.024933    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="87350580-849d-40cd-a8f6-618c56f16493" containerName="revoke-keys"
Oct 30 05:10:32.029767 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.029751    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:32.047379 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.047355    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"]
Oct 30 05:10:32.055843 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.055822    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:10:32.055993 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.055852    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:10:32.077890 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.077863    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45642b14-vhld2\" (UID: \"8824e440-c110-4c3d-a1bc-067d8032310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:32.078019 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.077970    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtjsj\" (UniqueName: \"kubernetes.io/projected/8824e440-c110-4c3d-a1bc-067d8032310a-kube-api-access-jtjsj\") pod \"maas-api-revoke-sub-45642b14-vhld2\" (UID: \"8824e440-c110-4c3d-a1bc-067d8032310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:32.078127 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.078110    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45642b14-vhld2\" (UID: \"8824e440-c110-4c3d-a1bc-067d8032310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:32.178833 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.178806    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtjsj\" (UniqueName: \"kubernetes.io/projected/8824e440-c110-4c3d-a1bc-067d8032310a-kube-api-access-jtjsj\") pod \"maas-api-revoke-sub-45642b14-vhld2\" (UID: \"8824e440-c110-4c3d-a1bc-067d8032310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:32.179011 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.178917    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45642b14-vhld2\" (UID: \"8824e440-c110-4c3d-a1bc-067d8032310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:32.179106 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.179019    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45642b14-vhld2\" (UID: \"8824e440-c110-4c3d-a1bc-067d8032310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:32.179732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.179664    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45642b14-vhld2\" (UID: \"8824e440-c110-4c3d-a1bc-067d8032310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:32.181696 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.181672    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45642b14-vhld2\" (UID: \"8824e440-c110-4c3d-a1bc-067d8032310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:32.203473 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.203444    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtjsj\" (UniqueName: \"kubernetes.io/projected/8824e440-c110-4c3d-a1bc-067d8032310a-kube-api-access-jtjsj\") pod \"maas-api-revoke-sub-45642b14-vhld2\" (UID: \"8824e440-c110-4c3d-a1bc-067d8032310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:32.343919 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.343835    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:32.655184 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:32.655105    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"]
Oct 30 05:10:32.655944 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:10:32.655913    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8824e440_c110_4c3d_a1bc_067d8032310a.slice/crio-a170f3e17ccd7c5b7c70502ea59136146e19b73916f03f1f25c2a06d4cfb0947 WatchSource:0}: Error finding container a170f3e17ccd7c5b7c70502ea59136146e19b73916f03f1f25c2a06d4cfb0947: Status 404 returned error can't find the container with id a170f3e17ccd7c5b7c70502ea59136146e19b73916f03f1f25c2a06d4cfb0947
Oct 30 05:10:33.186922 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:33.186889    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"]
Oct 30 05:10:33.220271 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:33.220188    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"]
Oct 30 05:10:33.433235 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:33.433194    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2" event={"ID":"8824e440-c110-4c3d-a1bc-067d8032310a","Type":"ContainerStarted","Data":"2296339ba877cfcd2edfc093bc78dfd58c0854d010d677807dc5f1f33f30ece9"}
Oct 30 05:10:33.433501 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:33.433261    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2" event={"ID":"8824e440-c110-4c3d-a1bc-067d8032310a","Type":"ContainerStarted","Data":"a170f3e17ccd7c5b7c70502ea59136146e19b73916f03f1f25c2a06d4cfb0947"}
Oct 30 05:10:33.479357 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:33.479248    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2" podStartSLOduration=2.479233211 podStartE2EDuration="2.479233211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:33.476291689 +0000 UTC m=+4413.217714810" watchObservedRunningTime="2026-10-30 05:10:33.479233211 +0000 UTC m=+4413.220656307"
Oct 30 05:10:34.443006 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:34.442968    5072 generic.go:356] "Generic (PLEG): container finished" podID="8824e440-c110-4c3d-a1bc-067d8032310a" containerID="2296339ba877cfcd2edfc093bc78dfd58c0854d010d677807dc5f1f33f30ece9" exitCode=0
Oct 30 05:10:34.443556 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:34.443056    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2" event={"ID":"8824e440-c110-4c3d-a1bc-067d8032310a","Type":"ContainerDied","Data":"2296339ba877cfcd2edfc093bc78dfd58c0854d010d677807dc5f1f33f30ece9"}
Oct 30 05:10:35.607873 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:35.607847    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:35.740250 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:35.740174    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-openshift-ca\") pod \"8824e440-c110-4c3d-a1bc-067d8032310a\" (UID: \"8824e440-c110-4c3d-a1bc-067d8032310a\") "
Oct 30 05:10:35.740413 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:35.740280    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8824e440-c110-4c3d-a1bc-067d8032310a-kube-api-access-jtjsj\" (UniqueName: \"kubernetes.io/projected/8824e440-c110-4c3d-a1bc-067d8032310a-kube-api-access-jtjsj\") pod \"8824e440-c110-4c3d-a1bc-067d8032310a\" (UID: \"8824e440-c110-4c3d-a1bc-067d8032310a\") "
Oct 30 05:10:35.740413 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:35.740333    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-xks-ca\") pod \"8824e440-c110-4c3d-a1bc-067d8032310a\" (UID: \"8824e440-c110-4c3d-a1bc-067d8032310a\") "
Oct 30 05:10:35.740563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:35.740541    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-openshift-ca" pod "8824e440-c110-4c3d-a1bc-067d8032310a" (UID: "8824e440-c110-4c3d-a1bc-067d8032310a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:35.740852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:35.740827    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:35.743201 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:35.743178    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-xks-ca" pod "8824e440-c110-4c3d-a1bc-067d8032310a" (UID: "8824e440-c110-4c3d-a1bc-067d8032310a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:35.743313 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:35.743229    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8824e440-c110-4c3d-a1bc-067d8032310a-kube-api-access-jtjsj" pod "8824e440-c110-4c3d-a1bc-067d8032310a" (UID: "8824e440-c110-4c3d-a1bc-067d8032310a"). InnerVolumeSpecName "kube-api-access-jtjsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:35.842228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:35.842199    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtjsj\" (UniqueName: \"kubernetes.io/projected/8824e440-c110-4c3d-a1bc-067d8032310a-kube-api-access-jtjsj\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:35.842228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:35.842227    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8824e440-c110-4c3d-a1bc-067d8032310a-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:36.033545 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:36.033447    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"]
Oct 30 05:10:36.036814 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:36.036789    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67407bb-cbjp5"]
Oct 30 05:10:36.427739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:36.427653    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="114daf05-9ccf-4e38-82ad-f5859354f86e" path="/var/lib/kubelet/pods/114daf05-9ccf-4e38-82ad-f5859354f86e/volumes"
Oct 30 05:10:36.467272 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:36.467229    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2" event={"ID":"8824e440-c110-4c3d-a1bc-067d8032310a","Type":"ContainerDied","Data":"a170f3e17ccd7c5b7c70502ea59136146e19b73916f03f1f25c2a06d4cfb0947"}
Oct 30 05:10:36.467592 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:36.467442    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"
Oct 30 05:10:36.467744 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:36.467561    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a170f3e17ccd7c5b7c70502ea59136146e19b73916f03f1f25c2a06d4cfb0947"
Oct 30 05:10:38.187708 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.187667    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw" podUID="ecbbac18-76cc-4c8b-93aa-2346668f419a" containerName="payload-processing" containerID="cri-o://694f26abe4dea62c8a1b6be425276949e3f181322712ef6675700a741391e614" gracePeriod=25
Oct 30 05:10:38.221146 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.221115    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt" podUID="5cb20def-2c1e-4df2-b96a-c743d5c37939" containerName="payload-pre-processing" containerID="cri-o://8ba3864a0c9fba27c0604f422e7f5c78af617c7b3ed80c310c68072798faa1d4" gracePeriod=25
Oct 30 05:10:38.491382 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.491348    5072 generic.go:356] "Generic (PLEG): container finished" podID="5cb20def-2c1e-4df2-b96a-c743d5c37939" containerID="8ba3864a0c9fba27c0604f422e7f5c78af617c7b3ed80c310c68072798faa1d4" exitCode=0
Oct 30 05:10:38.491519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.491500    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt" event={"ID":"5cb20def-2c1e-4df2-b96a-c743d5c37939","Type":"ContainerDied","Data":"8ba3864a0c9fba27c0604f422e7f5c78af617c7b3ed80c310c68072798faa1d4"}
Oct 30 05:10:38.496600 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.496570    5072 generic.go:356] "Generic (PLEG): container finished" podID="ecbbac18-76cc-4c8b-93aa-2346668f419a" containerID="694f26abe4dea62c8a1b6be425276949e3f181322712ef6675700a741391e614" exitCode=0
Oct 30 05:10:38.496740 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.496617    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw" event={"ID":"ecbbac18-76cc-4c8b-93aa-2346668f419a","Type":"ContainerDied","Data":"694f26abe4dea62c8a1b6be425276949e3f181322712ef6675700a741391e614"}
Oct 30 05:10:38.519569 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.519546    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"
Oct 30 05:10:38.555730 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.555708    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"
Oct 30 05:10:38.689854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.684283    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb20def-2c1e-4df2-b96a-c743d5c37939-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb20def-2c1e-4df2-b96a-c743d5c37939-plugins-config-volume\") pod \"5cb20def-2c1e-4df2-b96a-c743d5c37939\" (UID: \"5cb20def-2c1e-4df2-b96a-c743d5c37939\") "
Oct 30 05:10:38.689854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.684520    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecbbac18-76cc-4c8b-93aa-2346668f419a-kube-api-access-txd97\" (UniqueName: \"kubernetes.io/projected/ecbbac18-76cc-4c8b-93aa-2346668f419a-kube-api-access-txd97\") pod \"ecbbac18-76cc-4c8b-93aa-2346668f419a\" (UID: \"ecbbac18-76cc-4c8b-93aa-2346668f419a\") "
Oct 30 05:10:38.689854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.684563    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecbbac18-76cc-4c8b-93aa-2346668f419a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbbac18-76cc-4c8b-93aa-2346668f419a-plugins-config-volume\") pod \"ecbbac18-76cc-4c8b-93aa-2346668f419a\" (UID: \"ecbbac18-76cc-4c8b-93aa-2346668f419a\") "
Oct 30 05:10:38.689854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.684626    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb20def-2c1e-4df2-b96a-c743d5c37939-kube-api-access-86x22\" (UniqueName: \"kubernetes.io/projected/5cb20def-2c1e-4df2-b96a-c743d5c37939-kube-api-access-86x22\") pod \"5cb20def-2c1e-4df2-b96a-c743d5c37939\" (UID: \"5cb20def-2c1e-4df2-b96a-c743d5c37939\") "
Oct 30 05:10:38.689854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.686063    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbbac18-76cc-4c8b-93aa-2346668f419a-plugins-config-volume" pod "ecbbac18-76cc-4c8b-93aa-2346668f419a" (UID: "ecbbac18-76cc-4c8b-93aa-2346668f419a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:38.689854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.686257    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb20def-2c1e-4df2-b96a-c743d5c37939-plugins-config-volume" pod "5cb20def-2c1e-4df2-b96a-c743d5c37939" (UID: "5cb20def-2c1e-4df2-b96a-c743d5c37939"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:38.689854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.689374    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb20def-2c1e-4df2-b96a-c743d5c37939-kube-api-access-86x22" pod "5cb20def-2c1e-4df2-b96a-c743d5c37939" (UID: "5cb20def-2c1e-4df2-b96a-c743d5c37939"). InnerVolumeSpecName "kube-api-access-86x22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:38.690449 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.690429    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbbac18-76cc-4c8b-93aa-2346668f419a-kube-api-access-txd97" pod "ecbbac18-76cc-4c8b-93aa-2346668f419a" (UID: "ecbbac18-76cc-4c8b-93aa-2346668f419a"). InnerVolumeSpecName "kube-api-access-txd97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:38.786612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.786519    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txd97\" (UniqueName: \"kubernetes.io/projected/ecbbac18-76cc-4c8b-93aa-2346668f419a-kube-api-access-txd97\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:38.786612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.786551    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbbac18-76cc-4c8b-93aa-2346668f419a-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:38.786612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.786561    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86x22\" (UniqueName: \"kubernetes.io/projected/5cb20def-2c1e-4df2-b96a-c743d5c37939-kube-api-access-86x22\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:38.786612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:38.786570    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb20def-2c1e-4df2-b96a-c743d5c37939-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:39.505509 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:39.505432    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"
Oct 30 05:10:39.505509 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:39.505483    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt" event={"ID":"5cb20def-2c1e-4df2-b96a-c743d5c37939","Type":"ContainerDied","Data":"4b64895c9a52ee2b5cac062eacf9d0f836cc568b5680eae2d6a28b9008a8529d"}
Oct 30 05:10:39.506116 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:39.505536    5072 scope.go:122] "RemoveContainer" containerID="8ba3864a0c9fba27c0604f422e7f5c78af617c7b3ed80c310c68072798faa1d4"
Oct 30 05:10:39.507435 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:39.507416    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"
Oct 30 05:10:39.507536 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:39.507427    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw" event={"ID":"ecbbac18-76cc-4c8b-93aa-2346668f419a","Type":"ContainerDied","Data":"984e6e7d390d5c59e3ecb08e8e2c854faa82c286a5c0b220a093aa67df7e53a3"}
Oct 30 05:10:39.519246 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:39.519230    5072 scope.go:122] "RemoveContainer" containerID="694f26abe4dea62c8a1b6be425276949e3f181322712ef6675700a741391e614"
Oct 30 05:10:39.563919 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:39.563881    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"]
Oct 30 05:10:39.574040 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:39.574008    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5496290-c59bb6f6c-w4dgt"]
Oct 30 05:10:39.605669 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:39.605640    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"]
Oct 30 05:10:39.609378 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:39.609356    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b5496290-5dbd66898-zm9gw"]
Oct 30 05:10:39.942904 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:39.942877    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"
Oct 30 05:10:39.990000 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:39.989970    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"
Oct 30 05:10:40.103896 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.103813    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7d87cd-c93b-49d6-8319-af928d231319-kube-api-access-8cq5c\" (UniqueName: \"kubernetes.io/projected/ee7d87cd-c93b-49d6-8319-af928d231319-kube-api-access-8cq5c\") pod \"ee7d87cd-c93b-49d6-8319-af928d231319\" (UID: \"ee7d87cd-c93b-49d6-8319-af928d231319\") "
Oct 30 05:10:40.104069 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.103899    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee7d87cd-c93b-49d6-8319-af928d231319-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee7d87cd-c93b-49d6-8319-af928d231319-plugins-config-volume\") pod \"ee7d87cd-c93b-49d6-8319-af928d231319\" (UID: \"ee7d87cd-c93b-49d6-8319-af928d231319\") "
Oct 30 05:10:40.104069 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.103977    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620fa980-d930-44fe-b306-5db27ed9e675-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620fa980-d930-44fe-b306-5db27ed9e675-plugins-config-volume\") pod \"620fa980-d930-44fe-b306-5db27ed9e675\" (UID: \"620fa980-d930-44fe-b306-5db27ed9e675\") "
Oct 30 05:10:40.104069 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.104031    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620fa980-d930-44fe-b306-5db27ed9e675-kube-api-access-2csfr\" (UniqueName: \"kubernetes.io/projected/620fa980-d930-44fe-b306-5db27ed9e675-kube-api-access-2csfr\") pod \"620fa980-d930-44fe-b306-5db27ed9e675\" (UID: \"620fa980-d930-44fe-b306-5db27ed9e675\") "
Oct 30 05:10:40.104357 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.104331    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee7d87cd-c93b-49d6-8319-af928d231319-plugins-config-volume" pod "ee7d87cd-c93b-49d6-8319-af928d231319" (UID: "ee7d87cd-c93b-49d6-8319-af928d231319"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:40.104467 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.104441    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620fa980-d930-44fe-b306-5db27ed9e675-plugins-config-volume" pod "620fa980-d930-44fe-b306-5db27ed9e675" (UID: "620fa980-d930-44fe-b306-5db27ed9e675"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:40.104648 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.104630    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620fa980-d930-44fe-b306-5db27ed9e675-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:40.104702 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.104654    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee7d87cd-c93b-49d6-8319-af928d231319-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:40.106932 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.106906    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7d87cd-c93b-49d6-8319-af928d231319-kube-api-access-8cq5c" pod "ee7d87cd-c93b-49d6-8319-af928d231319" (UID: "ee7d87cd-c93b-49d6-8319-af928d231319"). InnerVolumeSpecName "kube-api-access-8cq5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:40.106932 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.106933    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620fa980-d930-44fe-b306-5db27ed9e675-kube-api-access-2csfr" pod "620fa980-d930-44fe-b306-5db27ed9e675" (UID: "620fa980-d930-44fe-b306-5db27ed9e675"). InnerVolumeSpecName "kube-api-access-2csfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:40.206045 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.206020    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2csfr\" (UniqueName: \"kubernetes.io/projected/620fa980-d930-44fe-b306-5db27ed9e675-kube-api-access-2csfr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:40.206144 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.206053    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cq5c\" (UniqueName: \"kubernetes.io/projected/ee7d87cd-c93b-49d6-8319-af928d231319-kube-api-access-8cq5c\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:40.411328 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.411240    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb20def-2c1e-4df2-b96a-c743d5c37939" path="/var/lib/kubelet/pods/5cb20def-2c1e-4df2-b96a-c743d5c37939/volumes"
Oct 30 05:10:40.411743 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.411720    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecbbac18-76cc-4c8b-93aa-2346668f419a" path="/var/lib/kubelet/pods/ecbbac18-76cc-4c8b-93aa-2346668f419a/volumes"
Oct 30 05:10:40.518594 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.518558    5072 generic.go:356] "Generic (PLEG): container finished" podID="ee7d87cd-c93b-49d6-8319-af928d231319" containerID="4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d" exitCode=0
Oct 30 05:10:40.518594 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.518590    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2" event={"ID":"ee7d87cd-c93b-49d6-8319-af928d231319","Type":"ContainerDied","Data":"4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d"}
Oct 30 05:10:40.519176 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.518611    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"
Oct 30 05:10:40.519176 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.518655    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2" event={"ID":"ee7d87cd-c93b-49d6-8319-af928d231319","Type":"ContainerDied","Data":"f8ec07035ef79658f5ad7a0f3d564cacdd1f7ec5fc0c4bc834b68bfeb9c76002"}
Oct 30 05:10:40.519176 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.518678    5072 scope.go:122] "RemoveContainer" containerID="4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d"
Oct 30 05:10:40.520287 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.520265    5072 generic.go:356] "Generic (PLEG): container finished" podID="620fa980-d930-44fe-b306-5db27ed9e675" containerID="00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a" exitCode=0
Oct 30 05:10:40.520435 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.520318    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"
Oct 30 05:10:40.520435 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.520350    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx" event={"ID":"620fa980-d930-44fe-b306-5db27ed9e675","Type":"ContainerDied","Data":"00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a"}
Oct 30 05:10:40.520435 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.520401    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx" event={"ID":"620fa980-d930-44fe-b306-5db27ed9e675","Type":"ContainerDied","Data":"0f22de327a9dc0218c6f0e10cbf20e3d1f599ac0cdd0017474c8438e940d87ff"}
Oct 30 05:10:40.530193 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.530176    5072 scope.go:122] "RemoveContainer" containerID="4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d"
Oct 30 05:10:40.530524 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:10:40.530504    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d\": container with ID starting with 4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d not found: ID does not exist" containerID="4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d"
Oct 30 05:10:40.530581 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.530534    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d"} err="failed to get container status \"4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d\": rpc error: code = NotFound desc = could not find container \"4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d\": container with ID starting with 4ee179a2ba51217507a2732ca64240cf6ba8c02eab734bae4a5c07bd068a8c9d not found: ID does not exist"
Oct 30 05:10:40.530581 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.530553    5072 scope.go:122] "RemoveContainer" containerID="00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a"
Oct 30 05:10:40.541355 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.541337    5072 scope.go:122] "RemoveContainer" containerID="00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a"
Oct 30 05:10:40.541650 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:10:40.541629    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a\": container with ID starting with 00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a not found: ID does not exist" containerID="00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a"
Oct 30 05:10:40.541720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.541662    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a"} err="failed to get container status \"00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a\": rpc error: code = NotFound desc = could not find container \"00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a\": container with ID starting with 00c5edbd3cd05fae3b36386bff0163e4245387de800e7f2048706e967f99281a not found: ID does not exist"
Oct 30 05:10:40.575660 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.575625    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"]
Oct 30 05:10:40.580482 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.580456    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-503804-8d5f44cf4-qfln2"]
Oct 30 05:10:40.614995 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.614965    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"]
Oct 30 05:10:40.619638 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:40.619605    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-503804-6558f98787-75dmx"]
Oct 30 05:10:41.752430 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:10:41.752379    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:10:42.410563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.410534    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620fa980-d930-44fe-b306-5db27ed9e675" path="/var/lib/kubelet/pods/620fa980-d930-44fe-b306-5db27ed9e675/volumes"
Oct 30 05:10:42.411061 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.411044    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee7d87cd-c93b-49d6-8319-af928d231319" path="/var/lib/kubelet/pods/ee7d87cd-c93b-49d6-8319-af928d231319/volumes"
Oct 30 05:10:42.545033 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.544377    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"]
Oct 30 05:10:42.545515 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545477    5072 cpu_manager.go:414] "containerMap: removing container" podUID="620fa980-d930-44fe-b306-5db27ed9e675" containerName="payload-pre-processing"
Oct 30 05:10:42.545515 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545505    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620fa980-d930-44fe-b306-5db27ed9e675" containerName="payload-pre-processing"
Oct 30 05:10:42.545732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545532    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ecbbac18-76cc-4c8b-93aa-2346668f419a" containerName="payload-processing"
Oct 30 05:10:42.545732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545540    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbbac18-76cc-4c8b-93aa-2346668f419a" containerName="payload-processing"
Oct 30 05:10:42.545732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545553    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ee7d87cd-c93b-49d6-8319-af928d231319" containerName="payload-processing"
Oct 30 05:10:42.545732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545560    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7d87cd-c93b-49d6-8319-af928d231319" containerName="payload-processing"
Oct 30 05:10:42.545732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545580    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5cb20def-2c1e-4df2-b96a-c743d5c37939" containerName="payload-pre-processing"
Oct 30 05:10:42.545732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545587    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb20def-2c1e-4df2-b96a-c743d5c37939" containerName="payload-pre-processing"
Oct 30 05:10:42.545732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545631    5072 cpu_manager.go:414] "containerMap: removing container" podUID="8824e440-c110-4c3d-a1bc-067d8032310a" containerName="revoke-keys"
Oct 30 05:10:42.545732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545639    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8824e440-c110-4c3d-a1bc-067d8032310a" containerName="revoke-keys"
Oct 30 05:10:42.546226 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545837    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="8824e440-c110-4c3d-a1bc-067d8032310a" containerName="revoke-keys"
Oct 30 05:10:42.546226 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545865    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="620fa980-d930-44fe-b306-5db27ed9e675" containerName="payload-pre-processing"
Oct 30 05:10:42.546226 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545877    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbbac18-76cc-4c8b-93aa-2346668f419a" containerName="payload-processing"
Oct 30 05:10:42.546226 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545888    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7d87cd-c93b-49d6-8319-af928d231319" containerName="payload-processing"
Oct 30 05:10:42.546226 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.545899    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb20def-2c1e-4df2-b96a-c743d5c37939" containerName="payload-pre-processing"
Oct 30 05:10:42.562784 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.562754    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"]
Oct 30 05:10:42.562957 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.562874    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:42.578277 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.578252    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:10:42.578448 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.578309    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:10:42.645830 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.645801    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89987aa0-k65b9\" (UID: \"c052089a-2b41-415e-ab0c-cacefb08d619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:42.645997 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.645920    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89987aa0-k65b9\" (UID: \"c052089a-2b41-415e-ab0c-cacefb08d619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:42.646048 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.646013    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq7ck\" (UniqueName: \"kubernetes.io/projected/c052089a-2b41-415e-ab0c-cacefb08d619-kube-api-access-pq7ck\") pod \"maas-api-revoke-sub-89987aa0-k65b9\" (UID: \"c052089a-2b41-415e-ab0c-cacefb08d619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:42.746856 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.746828    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89987aa0-k65b9\" (UID: \"c052089a-2b41-415e-ab0c-cacefb08d619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:42.746991 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.746910    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89987aa0-k65b9\" (UID: \"c052089a-2b41-415e-ab0c-cacefb08d619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:42.746991 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.746962    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7ck\" (UniqueName: \"kubernetes.io/projected/c052089a-2b41-415e-ab0c-cacefb08d619-kube-api-access-pq7ck\") pod \"maas-api-revoke-sub-89987aa0-k65b9\" (UID: \"c052089a-2b41-415e-ab0c-cacefb08d619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:42.747462 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.747444    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89987aa0-k65b9\" (UID: \"c052089a-2b41-415e-ab0c-cacefb08d619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:42.749495 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.749472    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89987aa0-k65b9\" (UID: \"c052089a-2b41-415e-ab0c-cacefb08d619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:42.771434 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.771403    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7ck\" (UniqueName: \"kubernetes.io/projected/c052089a-2b41-415e-ab0c-cacefb08d619-kube-api-access-pq7ck\") pod \"maas-api-revoke-sub-89987aa0-k65b9\" (UID: \"c052089a-2b41-415e-ab0c-cacefb08d619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:42.887797 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:42.887765    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:43.198035 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:10:43.198002    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc052089a_2b41_415e_ab0c_cacefb08d619.slice/crio-52eae12fd81c2eed92fc3f151effcfa6e38492d73377ec096b9b319341c564b6 WatchSource:0}: Error finding container 52eae12fd81c2eed92fc3f151effcfa6e38492d73377ec096b9b319341c564b6: Status 404 returned error can't find the container with id 52eae12fd81c2eed92fc3f151effcfa6e38492d73377ec096b9b319341c564b6
Oct 30 05:10:43.198423 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.198402    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"]
Oct 30 05:10:43.314815 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.314776    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"]
Oct 30 05:10:43.320528 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.320507    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:43.338459 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.338431    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"]
Oct 30 05:10:43.355567 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.355539    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97d86dd9-bhqtv\" (UID: \"f5602814-ae4e-4a87-8ed6-06f42807d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:43.355705 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.355579    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97d86dd9-bhqtv\" (UID: \"f5602814-ae4e-4a87-8ed6-06f42807d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:43.355773 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.355739    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkjbf\" (UniqueName: \"kubernetes.io/projected/f5602814-ae4e-4a87-8ed6-06f42807d690-kube-api-access-bkjbf\") pod \"maas-api-revoke-sub-97d86dd9-bhqtv\" (UID: \"f5602814-ae4e-4a87-8ed6-06f42807d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:43.457264 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.457183    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkjbf\" (UniqueName: \"kubernetes.io/projected/f5602814-ae4e-4a87-8ed6-06f42807d690-kube-api-access-bkjbf\") pod \"maas-api-revoke-sub-97d86dd9-bhqtv\" (UID: \"f5602814-ae4e-4a87-8ed6-06f42807d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:43.457447 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.457302    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97d86dd9-bhqtv\" (UID: \"f5602814-ae4e-4a87-8ed6-06f42807d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:43.457447 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.457337    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97d86dd9-bhqtv\" (UID: \"f5602814-ae4e-4a87-8ed6-06f42807d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:43.458028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.458002    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97d86dd9-bhqtv\" (UID: \"f5602814-ae4e-4a87-8ed6-06f42807d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:43.459996 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.459970    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97d86dd9-bhqtv\" (UID: \"f5602814-ae4e-4a87-8ed6-06f42807d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:43.483668 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.483642    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkjbf\" (UniqueName: \"kubernetes.io/projected/f5602814-ae4e-4a87-8ed6-06f42807d690-kube-api-access-bkjbf\") pod \"maas-api-revoke-sub-97d86dd9-bhqtv\" (UID: \"f5602814-ae4e-4a87-8ed6-06f42807d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:43.583374 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.583334    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9" event={"ID":"c052089a-2b41-415e-ab0c-cacefb08d619","Type":"ContainerStarted","Data":"1f2ab2daa4d56ee06d253ea3b5f0d7b67d8f4700e29a7d9accde3f33c9fe39b3"}
Oct 30 05:10:43.583550 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.583395    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9" event={"ID":"c052089a-2b41-415e-ab0c-cacefb08d619","Type":"ContainerStarted","Data":"52eae12fd81c2eed92fc3f151effcfa6e38492d73377ec096b9b319341c564b6"}
Oct 30 05:10:43.630007 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.629941    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9" podStartSLOduration=1.629921121 podStartE2EDuration="1.629921121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:43.624779746 +0000 UTC m=+4423.366202844" watchObservedRunningTime="2026-10-30 05:10:43.629921121 +0000 UTC m=+4423.371344220"
Oct 30 05:10:43.634771 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.634745    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:43.941886 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:43.940742    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"]
Oct 30 05:10:43.942868 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:10:43.942835    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5602814_ae4e_4a87_8ed6_06f42807d690.slice/crio-1847dae6da81510921a7088571b0876cb9524e07959c72845a9fdb1a960826e9 WatchSource:0}: Error finding container 1847dae6da81510921a7088571b0876cb9524e07959c72845a9fdb1a960826e9: Status 404 returned error can't find the container with id 1847dae6da81510921a7088571b0876cb9524e07959c72845a9fdb1a960826e9
Oct 30 05:10:44.593038 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:44.592943    5072 generic.go:356] "Generic (PLEG): container finished" podID="c052089a-2b41-415e-ab0c-cacefb08d619" containerID="1f2ab2daa4d56ee06d253ea3b5f0d7b67d8f4700e29a7d9accde3f33c9fe39b3" exitCode=0
Oct 30 05:10:44.593038 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:44.593022    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9" event={"ID":"c052089a-2b41-415e-ab0c-cacefb08d619","Type":"ContainerDied","Data":"1f2ab2daa4d56ee06d253ea3b5f0d7b67d8f4700e29a7d9accde3f33c9fe39b3"}
Oct 30 05:10:44.594868 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:44.594846    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv" event={"ID":"f5602814-ae4e-4a87-8ed6-06f42807d690","Type":"ContainerStarted","Data":"a05fe32b547d512fc3340b34a5757c28f9685a1b768be9ad084d81e5acb85709"}
Oct 30 05:10:44.595006 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:44.594882    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv" event={"ID":"f5602814-ae4e-4a87-8ed6-06f42807d690","Type":"ContainerStarted","Data":"1847dae6da81510921a7088571b0876cb9524e07959c72845a9fdb1a960826e9"}
Oct 30 05:10:44.682546 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:44.682492    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv" podStartSLOduration=1.682477722 podStartE2EDuration="1.682477722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:44.681375277 +0000 UTC m=+4424.422798391" watchObservedRunningTime="2026-10-30 05:10:44.682477722 +0000 UTC m=+4424.423900819"
Oct 30 05:10:45.604732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:45.604699    5072 generic.go:356] "Generic (PLEG): container finished" podID="f5602814-ae4e-4a87-8ed6-06f42807d690" containerID="a05fe32b547d512fc3340b34a5757c28f9685a1b768be9ad084d81e5acb85709" exitCode=0
Oct 30 05:10:45.605226 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:45.604778    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv" event={"ID":"f5602814-ae4e-4a87-8ed6-06f42807d690","Type":"ContainerDied","Data":"a05fe32b547d512fc3340b34a5757c28f9685a1b768be9ad084d81e5acb85709"}
Oct 30 05:10:45.788369 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:45.788339    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:45.887949 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:45.887868    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c052089a-2b41-415e-ab0c-cacefb08d619-kube-api-access-pq7ck\" (UniqueName: \"kubernetes.io/projected/c052089a-2b41-415e-ab0c-cacefb08d619-kube-api-access-pq7ck\") pod \"c052089a-2b41-415e-ab0c-cacefb08d619\" (UID: \"c052089a-2b41-415e-ab0c-cacefb08d619\") "
Oct 30 05:10:45.888161 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:45.887948    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-openshift-ca\") pod \"c052089a-2b41-415e-ab0c-cacefb08d619\" (UID: \"c052089a-2b41-415e-ab0c-cacefb08d619\") "
Oct 30 05:10:45.888161 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:45.888069    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-xks-ca\") pod \"c052089a-2b41-415e-ab0c-cacefb08d619\" (UID: \"c052089a-2b41-415e-ab0c-cacefb08d619\") "
Oct 30 05:10:45.888441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:45.888403    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-openshift-ca" pod "c052089a-2b41-415e-ab0c-cacefb08d619" (UID: "c052089a-2b41-415e-ab0c-cacefb08d619"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:45.888701 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:45.888676    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:45.890172 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:45.890141    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c052089a-2b41-415e-ab0c-cacefb08d619-kube-api-access-pq7ck" pod "c052089a-2b41-415e-ab0c-cacefb08d619" (UID: "c052089a-2b41-415e-ab0c-cacefb08d619"). InnerVolumeSpecName "kube-api-access-pq7ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:45.890276 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:45.890254    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-xks-ca" pod "c052089a-2b41-415e-ab0c-cacefb08d619" (UID: "c052089a-2b41-415e-ab0c-cacefb08d619"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:45.989494 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:45.989460    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq7ck\" (UniqueName: \"kubernetes.io/projected/c052089a-2b41-415e-ab0c-cacefb08d619-kube-api-access-pq7ck\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:45.989494 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:45.989491    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c052089a-2b41-415e-ab0c-cacefb08d619-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:46.613678 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:46.613648    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"
Oct 30 05:10:46.614198 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:46.613678    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9" event={"ID":"c052089a-2b41-415e-ab0c-cacefb08d619","Type":"ContainerDied","Data":"52eae12fd81c2eed92fc3f151effcfa6e38492d73377ec096b9b319341c564b6"}
Oct 30 05:10:46.614198 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:46.613725    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52eae12fd81c2eed92fc3f151effcfa6e38492d73377ec096b9b319341c564b6"
Oct 30 05:10:46.786216 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:46.786193    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:46.899593 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:46.899559    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-openshift-ca\") pod \"f5602814-ae4e-4a87-8ed6-06f42807d690\" (UID: \"f5602814-ae4e-4a87-8ed6-06f42807d690\") "
Oct 30 05:10:46.899760 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:46.899613    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-xks-ca\") pod \"f5602814-ae4e-4a87-8ed6-06f42807d690\" (UID: \"f5602814-ae4e-4a87-8ed6-06f42807d690\") "
Oct 30 05:10:46.899760 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:46.899699    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5602814-ae4e-4a87-8ed6-06f42807d690-kube-api-access-bkjbf\" (UniqueName: \"kubernetes.io/projected/f5602814-ae4e-4a87-8ed6-06f42807d690-kube-api-access-bkjbf\") pod \"f5602814-ae4e-4a87-8ed6-06f42807d690\" (UID: \"f5602814-ae4e-4a87-8ed6-06f42807d690\") "
Oct 30 05:10:46.899978 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:46.899951    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-openshift-ca" pod "f5602814-ae4e-4a87-8ed6-06f42807d690" (UID: "f5602814-ae4e-4a87-8ed6-06f42807d690"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:46.900134 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:46.900118    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:46.901875 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:46.901854    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-xks-ca" pod "f5602814-ae4e-4a87-8ed6-06f42807d690" (UID: "f5602814-ae4e-4a87-8ed6-06f42807d690"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:46.901977 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:46.901952    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5602814-ae4e-4a87-8ed6-06f42807d690-kube-api-access-bkjbf" pod "f5602814-ae4e-4a87-8ed6-06f42807d690" (UID: "f5602814-ae4e-4a87-8ed6-06f42807d690"). InnerVolumeSpecName "kube-api-access-bkjbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:47.001611 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:47.001581    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5602814-ae4e-4a87-8ed6-06f42807d690-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:47.001611 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:47.001610    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkjbf\" (UniqueName: \"kubernetes.io/projected/f5602814-ae4e-4a87-8ed6-06f42807d690-kube-api-access-bkjbf\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:47.622511 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:47.622471    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv" event={"ID":"f5602814-ae4e-4a87-8ed6-06f42807d690","Type":"ContainerDied","Data":"1847dae6da81510921a7088571b0876cb9524e07959c72845a9fdb1a960826e9"}
Oct 30 05:10:47.622511 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:47.622495    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"
Oct 30 05:10:47.622974 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:47.622530    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1847dae6da81510921a7088571b0876cb9524e07959c72845a9fdb1a960826e9"
Oct 30 05:10:48.822823 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:48.822781    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"]
Oct 30 05:10:48.823365 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:48.823034    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk" podUID="37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8" containerName="maas-api" containerID="cri-o://61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08" gracePeriod=30
Oct 30 05:10:49.162038 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.162010    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:10:49.229331 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.229293    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-maas-api-tls\") pod \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") "
Oct 30 05:10:49.229536 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.229459    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-metrics-certs\") pod \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") "
Oct 30 05:10:49.229536 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.229501    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-kube-api-access-rlm2g\" (UniqueName: \"kubernetes.io/projected/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-kube-api-access-rlm2g\") pod \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") "
Oct 30 05:10:49.229715 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.229682    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-openshift-service-ca\") pod \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\" (UID: \"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8\") "
Oct 30 05:10:49.230068 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.230040    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-openshift-service-ca" pod "37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8" (UID: "37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:49.230456 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.230432    5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:49.232096 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.232054    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-metrics-certs" pod "37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8" (UID: "37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:49.232212 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.232194    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-maas-api-tls" pod "37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8" (UID: "37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:49.234156 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.234127    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-kube-api-access-rlm2g" pod "37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8" (UID: "37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8"). InnerVolumeSpecName "kube-api-access-rlm2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:49.331825 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.331784    5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-metrics-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:49.331825 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.331813    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlm2g\" (UniqueName: \"kubernetes.io/projected/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-kube-api-access-rlm2g\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:49.331825 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.331823    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:49.646187 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.646152    5072 generic.go:356] "Generic (PLEG): container finished" podID="37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8" containerID="61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08" exitCode=0
Oct 30 05:10:49.646521 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.646208    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"
Oct 30 05:10:49.647158 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.646232    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk" event={"ID":"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8","Type":"ContainerDied","Data":"61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08"}
Oct 30 05:10:49.647277 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.647196    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk" event={"ID":"37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8","Type":"ContainerDied","Data":"33c511e7f752b5005e58cd8940727248bc3c0965543da9bfc29df13290ef6937"}
Oct 30 05:10:49.647277 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.647255    5072 scope.go:122] "RemoveContainer" containerID="61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08"
Oct 30 05:10:49.685349 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.685322    5072 scope.go:122] "RemoveContainer" containerID="61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08"
Oct 30 05:10:49.685705 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:10:49.685679    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08\": container with ID starting with 61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08 not found: ID does not exist" containerID="61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08"
Oct 30 05:10:49.685818 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.685717    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08"} err="failed to get container status \"61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08\": rpc error: code = NotFound desc = could not find container \"61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08\": container with ID starting with 61162ced14006ac3f4ad0dab13eb176ddb3ac4084f6c2416c663cd23e9487b08 not found: ID does not exist"
Oct 30 05:10:49.707645 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.707614    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"]
Oct 30 05:10:49.720601 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:49.720573    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5496290-6d4b4f7447-564rk"]
Oct 30 05:10:50.408724 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:50.408688    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8" path="/var/lib/kubelet/pods/37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8/volumes"
Oct 30 05:10:52.033415 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:52.033373    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"]
Oct 30 05:10:52.045964 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:52.045930    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b4b686-z5btt"]
Oct 30 05:10:52.409019 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:52.408944    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d426b5b-3875-48b9-8c8c-cc9deb7e68a7" path="/var/lib/kubelet/pods/8d426b5b-3875-48b9-8c8c-cc9deb7e68a7/volumes"
Oct 30 05:10:53.846007 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.845973    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"]
Oct 30 05:10:53.846864 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.846839    5072 cpu_manager.go:414] "containerMap: removing container" podUID="c052089a-2b41-415e-ab0c-cacefb08d619" containerName="revoke-keys"
Oct 30 05:10:53.847005 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.846865    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c052089a-2b41-415e-ab0c-cacefb08d619" containerName="revoke-keys"
Oct 30 05:10:53.847005 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.846938    5072 cpu_manager.go:414] "containerMap: removing container" podUID="37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8" containerName="maas-api"
Oct 30 05:10:53.847005 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.846948    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8" containerName="maas-api"
Oct 30 05:10:53.847005 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.846965    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f5602814-ae4e-4a87-8ed6-06f42807d690" containerName="revoke-keys"
Oct 30 05:10:53.847005 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.846973    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5602814-ae4e-4a87-8ed6-06f42807d690" containerName="revoke-keys"
Oct 30 05:10:53.847321 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.847167    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c052089a-2b41-415e-ab0c-cacefb08d619" containerName="revoke-keys"
Oct 30 05:10:53.847321 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.847199    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ea75c6-e6e6-4dc5-a1e7-2aa4c063d1f8" containerName="maas-api"
Oct 30 05:10:53.847321 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.847210    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5602814-ae4e-4a87-8ed6-06f42807d690" containerName="revoke-keys"
Oct 30 05:10:53.852543 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.852518    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:53.869270 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.869240    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"]
Oct 30 05:10:53.877456 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.877425    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:10:53.877625 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.877609    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:10:53.994516 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.994486    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2187543e-5mtqs\" (UID: \"d76256e9-dc69-471a-8542-59c49ccc5127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:53.994808 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.994778    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpgx\" (UniqueName: \"kubernetes.io/projected/d76256e9-dc69-471a-8542-59c49ccc5127-kube-api-access-rzpgx\") pod \"maas-api-revoke-sub-2187543e-5mtqs\" (UID: \"d76256e9-dc69-471a-8542-59c49ccc5127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:53.994938 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:53.994821    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2187543e-5mtqs\" (UID: \"d76256e9-dc69-471a-8542-59c49ccc5127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:54.034268 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:54.034229    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"]
Oct 30 05:10:54.037423 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:54.037397    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b49af0c-h69r4"]
Oct 30 05:10:54.096524 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:54.096439    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpgx\" (UniqueName: \"kubernetes.io/projected/d76256e9-dc69-471a-8542-59c49ccc5127-kube-api-access-rzpgx\") pod \"maas-api-revoke-sub-2187543e-5mtqs\" (UID: \"d76256e9-dc69-471a-8542-59c49ccc5127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:54.096524 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:54.096475    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2187543e-5mtqs\" (UID: \"d76256e9-dc69-471a-8542-59c49ccc5127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:54.096721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:54.096528    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2187543e-5mtqs\" (UID: \"d76256e9-dc69-471a-8542-59c49ccc5127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:54.097243 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:54.097219    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2187543e-5mtqs\" (UID: \"d76256e9-dc69-471a-8542-59c49ccc5127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:54.099410 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:54.099379    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2187543e-5mtqs\" (UID: \"d76256e9-dc69-471a-8542-59c49ccc5127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:54.121242 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:54.121215    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpgx\" (UniqueName: \"kubernetes.io/projected/d76256e9-dc69-471a-8542-59c49ccc5127-kube-api-access-rzpgx\") pod \"maas-api-revoke-sub-2187543e-5mtqs\" (UID: \"d76256e9-dc69-471a-8542-59c49ccc5127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:54.168206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:54.168179    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:54.408688 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:54.408610    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d205d118-eeda-400a-9338-a28f9aac7a61" path="/var/lib/kubelet/pods/d205d118-eeda-400a-9338-a28f9aac7a61/volumes"
Oct 30 05:10:54.478018 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:54.477989    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"]
Oct 30 05:10:54.478840 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:10:54.478817    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76256e9_dc69_471a_8542_59c49ccc5127.slice/crio-e7cc4dc3cbdd35971624c5be26df234f5a8a9eb40fbd1ac7f3acb92056dc7e31 WatchSource:0}: Error finding container e7cc4dc3cbdd35971624c5be26df234f5a8a9eb40fbd1ac7f3acb92056dc7e31: Status 404 returned error can't find the container with id e7cc4dc3cbdd35971624c5be26df234f5a8a9eb40fbd1ac7f3acb92056dc7e31
Oct 30 05:10:54.684188 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:54.684153    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs" event={"ID":"d76256e9-dc69-471a-8542-59c49ccc5127","Type":"ContainerStarted","Data":"e7cc4dc3cbdd35971624c5be26df234f5a8a9eb40fbd1ac7f3acb92056dc7e31"}
Oct 30 05:10:55.692891 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:55.692859    5072 generic.go:356] "Generic (PLEG): container finished" podID="d76256e9-dc69-471a-8542-59c49ccc5127" containerID="f3dfb053bacd6436fcb51e072502fb797747fce6a8849fa9d4726054aade09c5" exitCode=0
Oct 30 05:10:55.693384 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:55.692952    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs" event={"ID":"d76256e9-dc69-471a-8542-59c49ccc5127","Type":"ContainerDied","Data":"f3dfb053bacd6436fcb51e072502fb797747fce6a8849fa9d4726054aade09c5"}
Oct 30 05:10:56.864575 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:56.864549    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:57.035378 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:57.035292    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76256e9-dc69-471a-8542-59c49ccc5127-kube-api-access-rzpgx\" (UniqueName: \"kubernetes.io/projected/d76256e9-dc69-471a-8542-59c49ccc5127-kube-api-access-rzpgx\") pod \"d76256e9-dc69-471a-8542-59c49ccc5127\" (UID: \"d76256e9-dc69-471a-8542-59c49ccc5127\") "
Oct 30 05:10:57.035378 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:57.035373    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-xks-ca\") pod \"d76256e9-dc69-471a-8542-59c49ccc5127\" (UID: \"d76256e9-dc69-471a-8542-59c49ccc5127\") "
Oct 30 05:10:57.035585 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:57.035494    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-openshift-ca\") pod \"d76256e9-dc69-471a-8542-59c49ccc5127\" (UID: \"d76256e9-dc69-471a-8542-59c49ccc5127\") "
Oct 30 05:10:57.035870 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:57.035841    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-openshift-ca" pod "d76256e9-dc69-471a-8542-59c49ccc5127" (UID: "d76256e9-dc69-471a-8542-59c49ccc5127"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:57.036197 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:57.036111    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:57.037693 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:57.037669    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76256e9-dc69-471a-8542-59c49ccc5127-kube-api-access-rzpgx" pod "d76256e9-dc69-471a-8542-59c49ccc5127" (UID: "d76256e9-dc69-471a-8542-59c49ccc5127"). InnerVolumeSpecName "kube-api-access-rzpgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:57.037777 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:57.037722    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-xks-ca" pod "d76256e9-dc69-471a-8542-59c49ccc5127" (UID: "d76256e9-dc69-471a-8542-59c49ccc5127"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:57.137558 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:57.137520    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzpgx\" (UniqueName: \"kubernetes.io/projected/d76256e9-dc69-471a-8542-59c49ccc5127-kube-api-access-rzpgx\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:57.137558 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:57.137553    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76256e9-dc69-471a-8542-59c49ccc5127-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:10:57.709267 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:57.709232    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"
Oct 30 05:10:57.709267 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:57.709262    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs" event={"ID":"d76256e9-dc69-471a-8542-59c49ccc5127","Type":"ContainerDied","Data":"e7cc4dc3cbdd35971624c5be26df234f5a8a9eb40fbd1ac7f3acb92056dc7e31"}
Oct 30 05:10:57.709491 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:57.709306    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7cc4dc3cbdd35971624c5be26df234f5a8a9eb40fbd1ac7f3acb92056dc7e31"
Oct 30 05:10:58.964171 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:58.964135    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"]
Oct 30 05:10:58.964989 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:58.964965    5072 cpu_manager.go:414] "containerMap: removing container" podUID="d76256e9-dc69-471a-8542-59c49ccc5127" containerName="revoke-keys"
Oct 30 05:10:58.965130 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:58.964990    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76256e9-dc69-471a-8542-59c49ccc5127" containerName="revoke-keys"
Oct 30 05:10:58.965259 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:58.965242    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76256e9-dc69-471a-8542-59c49ccc5127" containerName="revoke-keys"
Oct 30 05:10:58.969214 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:58.969192    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:58.992736 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:58.992707    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"]
Oct 30 05:10:59.002510 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.002488    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-e6b837-openshift-default-dockercfg-9lwnd\""
Oct 30 05:10:59.162678 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.162647    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-istio-token\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.162846 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.162693    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-data\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.162846 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.162717    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9d8933-4567-4682-bbfa-5a459474528b-istio-podinfo\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.162927 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.162854    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-socket\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.162927 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.162884    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg75n\" (UniqueName: \"kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-kube-api-access-wg75n\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.163017 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.162995    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-certs\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.163174 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.163159    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-envoy\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.163218 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.163186    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-credential-socket\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.163218 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.163209    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9d8933-4567-4682-bbfa-5a459474528b-istiod-ca-cert\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.264030 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.263960    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-envoy\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.264030 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.263996    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-credential-socket\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.264030 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.264019    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9d8933-4567-4682-bbfa-5a459474528b-istiod-ca-cert\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.264338 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.264043    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-istio-token\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.264338 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.264110    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-data\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.264338 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.264138    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9d8933-4567-4682-bbfa-5a459474528b-istio-podinfo\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.264338 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.264171    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-socket\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.264338 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.264196    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg75n\" (UniqueName: \"kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-kube-api-access-wg75n\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.264338 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.264280    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-certs\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.264695 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.264457    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-credential-socket\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.264742 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.264686    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-data\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.264792 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.264762    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-socket\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.265052 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.265027    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-certs\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.265347 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.265326    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9d8933-4567-4682-bbfa-5a459474528b-istiod-ca-cert\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.266584 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.266564    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-envoy\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.266928 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.266906    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9d8933-4567-4682-bbfa-5a459474528b-istio-podinfo\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.290527 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.290492    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-istio-token\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.290731 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.290711    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg75n\" (UniqueName: \"kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-kube-api-access-wg75n\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.586438 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.586362    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:10:59.956661 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:10:59.956626    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9d8933_4567_4682_bbfa_5a459474528b.slice/crio-5cee50b8344942f573ca144ed0001cb18a67313fe7834c798d19f2fa442602ca WatchSource:0}: Error finding container 5cee50b8344942f573ca144ed0001cb18a67313fe7834c798d19f2fa442602ca: Status 404 returned error can't find the container with id 5cee50b8344942f573ca144ed0001cb18a67313fe7834c798d19f2fa442602ca
Oct 30 05:10:59.958700 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.958670    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"]
Oct 30 05:10:59.958912 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.958873    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:10:59.958996 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.958963    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:10:59.959061 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:10:59.959005    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:11:00.736964 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:00.736918    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h" event={"ID":"bf9d8933-4567-4682-bbfa-5a459474528b","Type":"ContainerStarted","Data":"2a0fa4127f57b38fc905b9e5b561a73e72676a7320c6e33fb82a0ca19d784242"}
Oct 30 05:11:00.737571 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:00.736989    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h" event={"ID":"bf9d8933-4567-4682-bbfa-5a459474528b","Type":"ContainerStarted","Data":"5cee50b8344942f573ca144ed0001cb18a67313fe7834c798d19f2fa442602ca"}
Oct 30 05:11:00.818540 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:00.818489    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h" podStartSLOduration=2.818473218 podStartE2EDuration="2.818473218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:00.814489891 +0000 UTC m=+4440.555912990" watchObservedRunningTime="2026-10-30 05:11:00.818473218 +0000 UTC m=+4440.559896312"
Oct 30 05:11:01.587336 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:01.587290    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:11:01.589521 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:01.589494    5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 05:11:01.589687 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:01.589548    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h" podUID="bf9d8933-4567-4682-bbfa-5a459474528b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 05:11:02.074319 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.074280    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"]
Oct 30 05:11:02.125330 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.125297    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"]
Oct 30 05:11:02.125564 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.125436    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.146747 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.146724    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e070ea\"/\"openshift-service-ca.crt\""
Oct 30 05:11:02.147007 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.146972    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e070ea\"/\"rate-test-model-e070ea-kserve-self-signed-certs\""
Oct 30 05:11:02.186494 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.186463    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e070ea\"/\"kube-root-ca.crt\""
Oct 30 05:11:02.186945 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.186926    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e070ea\"/\"default-dockercfg-zx98d\""
Oct 30 05:11:02.216014 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.215975    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-home\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.216014 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.216013    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-model-cache\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.216273 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.216091    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-dshm\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.216342 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.216307    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-tmp-dir\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.216424 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.216396    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhhkz\" (UniqueName: \"kubernetes.io/projected/e7105916-eb76-4348-86d3-820f79627195-kube-api-access-jhhkz\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.216485 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.216469    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7105916-eb76-4348-86d3-820f79627195-tls-certs\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.317330 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.317292    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-dshm\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.317542 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.317426    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-tmp-dir\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.317542 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.317497    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhhkz\" (UniqueName: \"kubernetes.io/projected/e7105916-eb76-4348-86d3-820f79627195-kube-api-access-jhhkz\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.317542 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.317533    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7105916-eb76-4348-86d3-820f79627195-tls-certs\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.317721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.317603    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-home\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.317721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.317625    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-model-cache\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.318284 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.318255    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-home\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.318416 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.318396    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-model-cache\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.318500 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.318480    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-tmp-dir\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.320638 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.320610    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-dshm\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.321379 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.321353    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7105916-eb76-4348-86d3-820f79627195-tls-certs\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.354628 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.352799    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhhkz\" (UniqueName: \"kubernetes.io/projected/e7105916-eb76-4348-86d3-820f79627195-kube-api-access-jhhkz\") pod \"rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") " pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.401113 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.399148    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"]
Oct 30 05:11:02.401113 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.399407    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerName="istio-proxy" containerID="cri-o://8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648" gracePeriod=30
Oct 30 05:11:02.448691 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.448648    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:02.596112 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.594379    5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 05:11:02.596112 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.594470    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h" podUID="bf9d8933-4567-4682-bbfa-5a459474528b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 05:11:02.925736 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:11:02.925701    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7105916_eb76_4348_86d3_820f79627195.slice/crio-6c7572490d610c90113f62c5db9e411e867fcecc1e15e9bfe6b00fc3d843311f WatchSource:0}: Error finding container 6c7572490d610c90113f62c5db9e411e867fcecc1e15e9bfe6b00fc3d843311f: Status 404 returned error can't find the container with id 6c7572490d610c90113f62c5db9e411e867fcecc1e15e9bfe6b00fc3d843311f
Oct 30 05:11:02.926620 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:02.926587    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"]
Oct 30 05:11:03.199209 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.198839    5072 scope.go:122] "RemoveContainer" containerID="2912121b9a92b42a4abc68481925675f1e65d5080ac1c9d1813e434cfda03ada"
Oct 30 05:11:03.587124 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.587019    5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 05:11:03.587124 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.587065    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h" podUID="bf9d8933-4567-4682-bbfa-5a459474528b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 05:11:03.590573 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.590550    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"]
Oct 30 05:11:03.596544 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.596516    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:03.617206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.617177    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"]
Oct 30 05:11:03.623714 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.623660    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:11:03.623714 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.623675    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:11:03.643948 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.643919    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08f59bf4-847q2\" (UID: \"017e8d82-8226-48a8-b622-d7117c705e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:03.644123 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.644101    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxhp\" (UniqueName: \"kubernetes.io/projected/017e8d82-8226-48a8-b622-d7117c705e16-kube-api-access-tkxhp\") pod \"maas-api-revoke-sub-08f59bf4-847q2\" (UID: \"017e8d82-8226-48a8-b622-d7117c705e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:03.644313 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.644285    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08f59bf4-847q2\" (UID: \"017e8d82-8226-48a8-b622-d7117c705e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:03.748796 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.748755    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08f59bf4-847q2\" (UID: \"017e8d82-8226-48a8-b622-d7117c705e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:03.749017 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.748858    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08f59bf4-847q2\" (UID: \"017e8d82-8226-48a8-b622-d7117c705e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:03.749017 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.748932    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxhp\" (UniqueName: \"kubernetes.io/projected/017e8d82-8226-48a8-b622-d7117c705e16-kube-api-access-tkxhp\") pod \"maas-api-revoke-sub-08f59bf4-847q2\" (UID: \"017e8d82-8226-48a8-b622-d7117c705e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:03.750060 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.749875    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08f59bf4-847q2\" (UID: \"017e8d82-8226-48a8-b622-d7117c705e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:03.753394 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.752400    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08f59bf4-847q2\" (UID: \"017e8d82-8226-48a8-b622-d7117c705e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:03.780099 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.777140    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxhp\" (UniqueName: \"kubernetes.io/projected/017e8d82-8226-48a8-b622-d7117c705e16-kube-api-access-tkxhp\") pod \"maas-api-revoke-sub-08f59bf4-847q2\" (UID: \"017e8d82-8226-48a8-b622-d7117c705e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:03.802228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.802192    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl" event={"ID":"e7105916-eb76-4348-86d3-820f79627195","Type":"ContainerStarted","Data":"de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178"}
Oct 30 05:11:03.802421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.802236    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl" event={"ID":"e7105916-eb76-4348-86d3-820f79627195","Type":"ContainerStarted","Data":"6c7572490d610c90113f62c5db9e411e867fcecc1e15e9bfe6b00fc3d843311f"}
Oct 30 05:11:03.912331 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:03.912241    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:04.278103 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:04.274166    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl" podStartSLOduration=2.274145863 podStartE2EDuration="2.274145863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:03.86937243 +0000 UTC m=+4443.610795533" watchObservedRunningTime="2026-10-30 05:11:04.274145863 +0000 UTC m=+4444.015568962"
Oct 30 05:11:04.278103 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:04.275925    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"]
Oct 30 05:11:04.292477 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:11:04.292448    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017e8d82_8226_48a8_b622_d7117c705e16.slice/crio-370588cc7eb07546989994f9501e795af1db7f6b553834ee6b0fcd363e6eb617 WatchSource:0}: Error finding container 370588cc7eb07546989994f9501e795af1db7f6b553834ee6b0fcd363e6eb617: Status 404 returned error can't find the container with id 370588cc7eb07546989994f9501e795af1db7f6b553834ee6b0fcd363e6eb617
Oct 30 05:11:04.449611 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:04.449578    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:04.588130 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:04.587752    5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 05:11:04.588130 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:04.587808    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h" podUID="bf9d8933-4567-4682-bbfa-5a459474528b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 05:11:04.825612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:04.825570    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2" event={"ID":"017e8d82-8226-48a8-b622-d7117c705e16","Type":"ContainerStarted","Data":"5cbe8b370bf24c87f6b777b8a24b9d535a97fb658c6a2cef4719f6bd93fd93ec"}
Oct 30 05:11:04.825798 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:04.825678    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2" event={"ID":"017e8d82-8226-48a8-b622-d7117c705e16","Type":"ContainerStarted","Data":"370588cc7eb07546989994f9501e795af1db7f6b553834ee6b0fcd363e6eb617"}
Oct 30 05:11:04.869159 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:04.867789    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2" podStartSLOduration=1.8677737589999999 podStartE2EDuration="1.867773759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:04.867464531 +0000 UTC m=+4444.608887630" watchObservedRunningTime="2026-10-30 05:11:04.867773759 +0000 UTC m=+4444.609196857"
Oct 30 05:11:05.586672 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:05.586641    5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 05:11:05.587243 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:05.586691    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h" podUID="bf9d8933-4567-4682-bbfa-5a459474528b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 05:11:05.835333 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:05.835296    5072 generic.go:356] "Generic (PLEG): container finished" podID="017e8d82-8226-48a8-b622-d7117c705e16" containerID="5cbe8b370bf24c87f6b777b8a24b9d535a97fb658c6a2cef4719f6bd93fd93ec" exitCode=0
Oct 30 05:11:05.835545 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:05.835432    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2" event={"ID":"017e8d82-8226-48a8-b622-d7117c705e16","Type":"ContainerDied","Data":"5cbe8b370bf24c87f6b777b8a24b9d535a97fb658c6a2cef4719f6bd93fd93ec"}
Oct 30 05:11:06.586665 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:06.586631    5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 05:11:06.586869 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:06.586683    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h" podUID="bf9d8933-4567-4682-bbfa-5a459474528b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 05:11:07.029546 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.029515    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:07.112853 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.112753    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017e8d82-8226-48a8-b622-d7117c705e16-kube-api-access-tkxhp\" (UniqueName: \"kubernetes.io/projected/017e8d82-8226-48a8-b622-d7117c705e16-kube-api-access-tkxhp\") pod \"017e8d82-8226-48a8-b622-d7117c705e16\" (UID: \"017e8d82-8226-48a8-b622-d7117c705e16\") "
Oct 30 05:11:07.113186 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.113157    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-openshift-ca\") pod \"017e8d82-8226-48a8-b622-d7117c705e16\" (UID: \"017e8d82-8226-48a8-b622-d7117c705e16\") "
Oct 30 05:11:07.113384 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.113362    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-xks-ca\") pod \"017e8d82-8226-48a8-b622-d7117c705e16\" (UID: \"017e8d82-8226-48a8-b622-d7117c705e16\") "
Oct 30 05:11:07.113538 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.113511    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-openshift-ca" pod "017e8d82-8226-48a8-b622-d7117c705e16" (UID: "017e8d82-8226-48a8-b622-d7117c705e16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:11:07.113962 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.113935    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:07.115880 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.115853    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017e8d82-8226-48a8-b622-d7117c705e16-kube-api-access-tkxhp" pod "017e8d82-8226-48a8-b622-d7117c705e16" (UID: "017e8d82-8226-48a8-b622-d7117c705e16"). InnerVolumeSpecName "kube-api-access-tkxhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:07.116782 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.116756    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-xks-ca" pod "017e8d82-8226-48a8-b622-d7117c705e16" (UID: "017e8d82-8226-48a8-b622-d7117c705e16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:11:07.215731 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.215694    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkxhp\" (UniqueName: \"kubernetes.io/projected/017e8d82-8226-48a8-b622-d7117c705e16-kube-api-access-tkxhp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:07.215731 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.215724    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017e8d82-8226-48a8-b622-d7117c705e16-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:07.825680 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.825649    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 05:11:07.855866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.855827    5072 generic.go:356] "Generic (PLEG): container finished" podID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerID="8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648" exitCode=0
Oct 30 05:11:07.856059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.855875    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"
Oct 30 05:11:07.856059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.855913    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" event={"ID":"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9","Type":"ContainerDied","Data":"8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648"}
Oct 30 05:11:07.856059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.855966    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9" event={"ID":"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9","Type":"ContainerDied","Data":"42446a50d1a1fbcef9cbc8fa78826a72883e810227edf1f9357c86e21858f95f"}
Oct 30 05:11:07.856059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.855998    5072 scope.go:122] "RemoveContainer" containerID="8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648"
Oct 30 05:11:07.858279 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.857774    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2" event={"ID":"017e8d82-8226-48a8-b622-d7117c705e16","Type":"ContainerDied","Data":"370588cc7eb07546989994f9501e795af1db7f6b553834ee6b0fcd363e6eb617"}
Oct 30 05:11:07.858279 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.857807    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="370588cc7eb07546989994f9501e795af1db7f6b553834ee6b0fcd363e6eb617"
Oct 30 05:11:07.858279 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.857852    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"
Oct 30 05:11:07.874320 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.873575    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"]
Oct 30 05:11:07.877634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.877612    5072 scope.go:122] "RemoveContainer" containerID="84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0"
Oct 30 05:11:07.964502 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.964461    5072 scope.go:122] "RemoveContainer" containerID="8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648"
Oct 30 05:11:07.965267 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:11:07.965228    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648\": container with ID starting with 8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648 not found: ID does not exist" containerID="8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648"
Oct 30 05:11:07.965420 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.965286    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648"} err="failed to get container status \"8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648\": rpc error: code = NotFound desc = could not find container \"8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648\": container with ID starting with 8d53fc0769cfa696ea1423f8016c81cb9c5dc966043e6985dda48ef17a724648 not found: ID does not exist"
Oct 30 05:11:07.965420 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.965316    5072 scope.go:122] "RemoveContainer" containerID="84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0"
Oct 30 05:11:07.965697 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:11:07.965677    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0\": container with ID starting with 84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0 not found: ID does not exist" containerID="84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0"
Oct 30 05:11:07.965767 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:07.965708    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0"} err="failed to get container status \"84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0\": rpc error: code = NotFound desc = could not find container \"84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0\": container with ID starting with 84f8d59893666c88db38f8c590e80d627d457a3aab9ac2e7f5c2a7abec6b99e0 not found: ID does not exist"
Oct 30 05:11:08.040908 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.040880    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-certs\") pod \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") "
Oct 30 05:11:08.041135 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.040919    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-kube-api-access-mnk8w\" (UniqueName: \"kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-kube-api-access-mnk8w\") pod \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") "
Oct 30 05:11:08.041135 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.041053    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-token\" (UniqueName: \"kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-token\") pod \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") "
Oct 30 05:11:08.041135 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.041097    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-credential-socket\") pod \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") "
Oct 30 05:11:08.041359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.041160    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istiod-ca-cert\") pod \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") "
Oct 30 05:11:08.041359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.041232    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-socket\") pod \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") "
Oct 30 05:11:08.041359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.041261    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-certs" pod "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" (UID: "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:08.041359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.041294    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-podinfo\") pod \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") "
Oct 30 05:11:08.041589 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.041364    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-data\") pod \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") "
Oct 30 05:11:08.041589 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.041404    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-envoy\") pod \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\" (UID: \"4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9\") "
Oct 30 05:11:08.041589 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.041452    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-credential-socket" pod "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" (UID: "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:08.041589 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.041541    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-socket" pod "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" (UID: "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:08.041888 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.041598    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istiod-ca-cert" pod "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" (UID: "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:11:08.041888 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.041682    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-data" pod "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" (UID: "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:08.042294 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.042227    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:08.042294 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.042255    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:08.042294 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.042271    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:08.042294 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.042289    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:08.042294 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.042304    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:08.043582 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.043547    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-token" pod "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" (UID: "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:08.043582 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.043574    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-kube-api-access-mnk8w" pod "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" (UID: "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9"). InnerVolumeSpecName "kube-api-access-mnk8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:08.043732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.043643    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-podinfo" pod "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" (UID: "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:11:08.043919 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.043904    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-envoy" pod "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" (UID: "4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:08.143512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.143426    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:08.143512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.143455    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:08.143512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.143464    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnk8w\" (UniqueName: \"kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-kube-api-access-mnk8w\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:08.143512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.143473    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:08.206734 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.206660    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"]
Oct 30 05:11:08.210381 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.210354    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-503804-openshift-default-5799fcdf7-2t2f9"]
Oct 30 05:11:08.274641 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.274613    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:11:08.275143 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.275114    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:11:08.276638 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.276609    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:11:08.412474 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:08.412366    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" path="/var/lib/kubelet/pods/4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9/volumes"
Oct 30 05:11:10.035223 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:10.035139    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"]
Oct 30 05:11:10.047951 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:10.047919    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac28b54c-2cntk"]
Oct 30 05:11:10.409020 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:10.408947    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5806f630-df78-471c-a399-e3c6413f370b" path="/var/lib/kubelet/pods/5806f630-df78-471c-a399-e3c6413f370b/volumes"
Oct 30 05:11:11.753051 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:11:11.753004    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:11:12.449394 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:12.449360    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:12.462404 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:12.462379    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:12.908638 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:12.908565    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:11:16.623639 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.623604    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"]
Oct 30 05:11:16.624329 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.624307    5072 cpu_manager.go:414] "containerMap: removing container" podUID="017e8d82-8226-48a8-b622-d7117c705e16" containerName="revoke-keys"
Oct 30 05:11:16.624329 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.624328    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="017e8d82-8226-48a8-b622-d7117c705e16" containerName="revoke-keys"
Oct 30 05:11:16.624509 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.624356    5072 cpu_manager.go:414] "containerMap: removing container" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerName="istio-proxy"
Oct 30 05:11:16.624509 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.624361    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerName="istio-proxy"
Oct 30 05:11:16.624509 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.624387    5072 cpu_manager.go:414] "containerMap: removing container" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerName="istio-proxy"
Oct 30 05:11:16.624509 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.624392    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerName="istio-proxy"
Oct 30 05:11:16.624647 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.624535    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerName="istio-proxy"
Oct 30 05:11:16.624647 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.624554    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="017e8d82-8226-48a8-b622-d7117c705e16" containerName="revoke-keys"
Oct 30 05:11:16.629359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.629336    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:16.645527 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.645497    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a202e63-k2nf5\" (UID: \"34cbb344-363e-4b10-91cf-660d41431ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:16.645662 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.645638    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a202e63-k2nf5\" (UID: \"34cbb344-363e-4b10-91cf-660d41431ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:16.645727 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.645700    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmhbg\" (UniqueName: \"kubernetes.io/projected/34cbb344-363e-4b10-91cf-660d41431ccd-kube-api-access-fmhbg\") pod \"maas-api-revoke-sub-5a202e63-k2nf5\" (UID: \"34cbb344-363e-4b10-91cf-660d41431ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:16.647108 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.647067    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"]
Oct 30 05:11:16.655619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.655598    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:11:16.655739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.655619    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:11:16.746944 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.746915    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a202e63-k2nf5\" (UID: \"34cbb344-363e-4b10-91cf-660d41431ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:16.747163 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.746986    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a202e63-k2nf5\" (UID: \"34cbb344-363e-4b10-91cf-660d41431ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:16.747163 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.747013    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmhbg\" (UniqueName: \"kubernetes.io/projected/34cbb344-363e-4b10-91cf-660d41431ccd-kube-api-access-fmhbg\") pod \"maas-api-revoke-sub-5a202e63-k2nf5\" (UID: \"34cbb344-363e-4b10-91cf-660d41431ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:16.747599 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.747580    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a202e63-k2nf5\" (UID: \"34cbb344-363e-4b10-91cf-660d41431ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:16.749542 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.749518    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a202e63-k2nf5\" (UID: \"34cbb344-363e-4b10-91cf-660d41431ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:16.771341 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.771314    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmhbg\" (UniqueName: \"kubernetes.io/projected/34cbb344-363e-4b10-91cf-660d41431ccd-kube-api-access-fmhbg\") pod \"maas-api-revoke-sub-5a202e63-k2nf5\" (UID: \"34cbb344-363e-4b10-91cf-660d41431ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:16.943451 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:16.943421    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:17.253906 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:11:17.253872    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34cbb344_363e_4b10_91cf_660d41431ccd.slice/crio-137ec91e29f05a697ca28ad74f44ebb8aa6850bf94bb7088ee928304b18e6f30 WatchSource:0}: Error finding container 137ec91e29f05a697ca28ad74f44ebb8aa6850bf94bb7088ee928304b18e6f30: Status 404 returned error can't find the container with id 137ec91e29f05a697ca28ad74f44ebb8aa6850bf94bb7088ee928304b18e6f30
Oct 30 05:11:17.254597 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:17.254574    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"]
Oct 30 05:11:17.937475 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:17.937441    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5" event={"ID":"34cbb344-363e-4b10-91cf-660d41431ccd","Type":"ContainerStarted","Data":"42142b98d62d07177bb41c26d5081ab3e1c5bf627f17d5dd011f039d74bab3ae"}
Oct 30 05:11:17.937897 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:17.937502    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5" event={"ID":"34cbb344-363e-4b10-91cf-660d41431ccd","Type":"ContainerStarted","Data":"137ec91e29f05a697ca28ad74f44ebb8aa6850bf94bb7088ee928304b18e6f30"}
Oct 30 05:11:17.999667 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:17.999618    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5" podStartSLOduration=1.999604269 podStartE2EDuration="1.999604269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:17.996183445 +0000 UTC m=+4457.737606543" watchObservedRunningTime="2026-10-30 05:11:17.999604269 +0000 UTC m=+4457.741027365"
Oct 30 05:11:18.946730 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:18.946688    5072 generic.go:356] "Generic (PLEG): container finished" podID="34cbb344-363e-4b10-91cf-660d41431ccd" containerID="42142b98d62d07177bb41c26d5081ab3e1c5bf627f17d5dd011f039d74bab3ae" exitCode=0
Oct 30 05:11:18.947212 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:18.946776    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5" event={"ID":"34cbb344-363e-4b10-91cf-660d41431ccd","Type":"ContainerDied","Data":"42142b98d62d07177bb41c26d5081ab3e1c5bf627f17d5dd011f039d74bab3ae"}
Oct 30 05:11:20.139965 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.139930    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:20.194588 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.194554    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34cbb344-363e-4b10-91cf-660d41431ccd-kube-api-access-fmhbg\" (UniqueName: \"kubernetes.io/projected/34cbb344-363e-4b10-91cf-660d41431ccd-kube-api-access-fmhbg\") pod \"34cbb344-363e-4b10-91cf-660d41431ccd\" (UID: \"34cbb344-363e-4b10-91cf-660d41431ccd\") "
Oct 30 05:11:20.195015 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.194987    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-xks-ca\") pod \"34cbb344-363e-4b10-91cf-660d41431ccd\" (UID: \"34cbb344-363e-4b10-91cf-660d41431ccd\") "
Oct 30 05:11:20.195275 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.195257    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-openshift-ca\") pod \"34cbb344-363e-4b10-91cf-660d41431ccd\" (UID: \"34cbb344-363e-4b10-91cf-660d41431ccd\") "
Oct 30 05:11:20.196342 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.196319    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-openshift-ca" pod "34cbb344-363e-4b10-91cf-660d41431ccd" (UID: "34cbb344-363e-4b10-91cf-660d41431ccd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:11:20.199349 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.199326    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34cbb344-363e-4b10-91cf-660d41431ccd-kube-api-access-fmhbg" pod "34cbb344-363e-4b10-91cf-660d41431ccd" (UID: "34cbb344-363e-4b10-91cf-660d41431ccd"). InnerVolumeSpecName "kube-api-access-fmhbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:20.205649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.205596    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-xks-ca" pod "34cbb344-363e-4b10-91cf-660d41431ccd" (UID: "34cbb344-363e-4b10-91cf-660d41431ccd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:11:20.297234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.297201    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmhbg\" (UniqueName: \"kubernetes.io/projected/34cbb344-363e-4b10-91cf-660d41431ccd-kube-api-access-fmhbg\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:20.297234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.297231    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:20.297234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.297240    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cbb344-363e-4b10-91cf-660d41431ccd-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:20.962444 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.962413    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"
Oct 30 05:11:20.962631 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.962437    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5" event={"ID":"34cbb344-363e-4b10-91cf-660d41431ccd","Type":"ContainerDied","Data":"137ec91e29f05a697ca28ad74f44ebb8aa6850bf94bb7088ee928304b18e6f30"}
Oct 30 05:11:20.962631 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:20.962488    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="137ec91e29f05a697ca28ad74f44ebb8aa6850bf94bb7088ee928304b18e6f30"
Oct 30 05:11:22.900351 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:22.900319    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"]
Oct 30 05:11:22.901091 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:22.901055    5072 cpu_manager.go:414] "containerMap: removing container" podUID="34cbb344-363e-4b10-91cf-660d41431ccd" containerName="revoke-keys"
Oct 30 05:11:22.901091 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:22.901087    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34cbb344-363e-4b10-91cf-660d41431ccd" containerName="revoke-keys"
Oct 30 05:11:22.901273 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:22.901226    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf86bd0-b7e9-43ae-a7e3-ac5841b92af9" containerName="istio-proxy"
Oct 30 05:11:22.901273 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:22.901235    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="34cbb344-363e-4b10-91cf-660d41431ccd" containerName="revoke-keys"
Oct 30 05:11:22.905091 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:22.905064    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:22.923066 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:22.923037    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"]
Oct 30 05:11:22.931576 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:22.931549    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:11:22.931754 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:22.931708    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:11:22.950678 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:22.950644    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb" podUID="ce161e2f-5b10-4d42-971e-0d51c68be921" containerName="main" containerID="cri-o://6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8" gracePeriod=45
Oct 30 05:11:23.032655 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.032625    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e311ac33-n99fv\" (UID: \"2446496a-8472-4ded-b6f5-b4ef56f518b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:23.032783 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.032739    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rs45\" (UniqueName: \"kubernetes.io/projected/2446496a-8472-4ded-b6f5-b4ef56f518b7-kube-api-access-2rs45\") pod \"maas-api-revoke-sub-e311ac33-n99fv\" (UID: \"2446496a-8472-4ded-b6f5-b4ef56f518b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:23.032853 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.032837    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e311ac33-n99fv\" (UID: \"2446496a-8472-4ded-b6f5-b4ef56f518b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:23.133979 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.133948    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e311ac33-n99fv\" (UID: \"2446496a-8472-4ded-b6f5-b4ef56f518b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:23.134226 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.134122    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e311ac33-n99fv\" (UID: \"2446496a-8472-4ded-b6f5-b4ef56f518b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:23.134226 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.134210    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rs45\" (UniqueName: \"kubernetes.io/projected/2446496a-8472-4ded-b6f5-b4ef56f518b7-kube-api-access-2rs45\") pod \"maas-api-revoke-sub-e311ac33-n99fv\" (UID: \"2446496a-8472-4ded-b6f5-b4ef56f518b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:23.134916 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.134888    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e311ac33-n99fv\" (UID: \"2446496a-8472-4ded-b6f5-b4ef56f518b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:23.138673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.138645    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e311ac33-n99fv\" (UID: \"2446496a-8472-4ded-b6f5-b4ef56f518b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:23.165780 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.165690    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rs45\" (UniqueName: \"kubernetes.io/projected/2446496a-8472-4ded-b6f5-b4ef56f518b7-kube-api-access-2rs45\") pod \"maas-api-revoke-sub-e311ac33-n99fv\" (UID: \"2446496a-8472-4ded-b6f5-b4ef56f518b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:23.220204 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.220173    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:23.242289 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.242262    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 05:11:23.336466 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.336428    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-tmp-dir\") pod \"ce161e2f-5b10-4d42-971e-0d51c68be921\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") "
Oct 30 05:11:23.336644 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.336522    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce161e2f-5b10-4d42-971e-0d51c68be921-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce161e2f-5b10-4d42-971e-0d51c68be921-tls-certs\") pod \"ce161e2f-5b10-4d42-971e-0d51c68be921\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") "
Oct 30 05:11:23.336644 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.336611    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce161e2f-5b10-4d42-971e-0d51c68be921-kube-api-access-v97cj\" (UniqueName: \"kubernetes.io/projected/ce161e2f-5b10-4d42-971e-0d51c68be921-kube-api-access-v97cj\") pod \"ce161e2f-5b10-4d42-971e-0d51c68be921\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") "
Oct 30 05:11:23.336789 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.336706    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-dshm\") pod \"ce161e2f-5b10-4d42-971e-0d51c68be921\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") "
Oct 30 05:11:23.336789 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.336729    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-tmp-dir" pod "ce161e2f-5b10-4d42-971e-0d51c68be921" (UID: "ce161e2f-5b10-4d42-971e-0d51c68be921"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:23.336891 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.336805    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-home\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-home\") pod \"ce161e2f-5b10-4d42-971e-0d51c68be921\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") "
Oct 30 05:11:23.336891 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.336877    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-model-cache\") pod \"ce161e2f-5b10-4d42-971e-0d51c68be921\" (UID: \"ce161e2f-5b10-4d42-971e-0d51c68be921\") "
Oct 30 05:11:23.337466 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.337416    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-model-cache" pod "ce161e2f-5b10-4d42-971e-0d51c68be921" (UID: "ce161e2f-5b10-4d42-971e-0d51c68be921"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:23.337630 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.337494    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.337807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.337764    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-home" pod "ce161e2f-5b10-4d42-971e-0d51c68be921" (UID: "ce161e2f-5b10-4d42-971e-0d51c68be921"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:23.339311 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.339284    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce161e2f-5b10-4d42-971e-0d51c68be921-tls-certs" pod "ce161e2f-5b10-4d42-971e-0d51c68be921" (UID: "ce161e2f-5b10-4d42-971e-0d51c68be921"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:11:23.339493 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.339468    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-dshm" pod "ce161e2f-5b10-4d42-971e-0d51c68be921" (UID: "ce161e2f-5b10-4d42-971e-0d51c68be921"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:23.339616 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.339506    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce161e2f-5b10-4d42-971e-0d51c68be921-kube-api-access-v97cj" pod "ce161e2f-5b10-4d42-971e-0d51c68be921" (UID: "ce161e2f-5b10-4d42-971e-0d51c68be921"). InnerVolumeSpecName "kube-api-access-v97cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:23.438915 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.438885    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce161e2f-5b10-4d42-971e-0d51c68be921-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.438915 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.438914    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v97cj\" (UniqueName: \"kubernetes.io/projected/ce161e2f-5b10-4d42-971e-0d51c68be921-kube-api-access-v97cj\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.439210 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.438923    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.439210 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.438931    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.439210 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.438939    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce161e2f-5b10-4d42-971e-0d51c68be921-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.531419 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:11:23.531388    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2446496a_8472_4ded_b6f5_b4ef56f518b7.slice/crio-7a50cef3944cfb12999130a5023d8a65ec481be5e448fc8c6b2ab25d576596c8 WatchSource:0}: Error finding container 7a50cef3944cfb12999130a5023d8a65ec481be5e448fc8c6b2ab25d576596c8: Status 404 returned error can't find the container with id 7a50cef3944cfb12999130a5023d8a65ec481be5e448fc8c6b2ab25d576596c8
Oct 30 05:11:23.533502 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.533476    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"]
Oct 30 05:11:23.985978 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.985934    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv" event={"ID":"2446496a-8472-4ded-b6f5-b4ef56f518b7","Type":"ContainerStarted","Data":"0e9c9bb13179ebc9703849e7819952774dc4ec503cd221c760325831a031638d"}
Oct 30 05:11:23.986441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.985995    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv" event={"ID":"2446496a-8472-4ded-b6f5-b4ef56f518b7","Type":"ContainerStarted","Data":"7a50cef3944cfb12999130a5023d8a65ec481be5e448fc8c6b2ab25d576596c8"}
Oct 30 05:11:23.987470 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.987445    5072 generic.go:356] "Generic (PLEG): container finished" podID="ce161e2f-5b10-4d42-971e-0d51c68be921" containerID="6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8" exitCode=0
Oct 30 05:11:23.987619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.987485    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb" event={"ID":"ce161e2f-5b10-4d42-971e-0d51c68be921","Type":"ContainerDied","Data":"6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8"}
Oct 30 05:11:23.987619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.987486    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"
Oct 30 05:11:23.987619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.987518    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb" event={"ID":"ce161e2f-5b10-4d42-971e-0d51c68be921","Type":"ContainerDied","Data":"07fd00730767e19b5f130763f13e818785b778d312c92c09d68dc5f533d833f4"}
Oct 30 05:11:23.987619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.987535    5072 scope.go:122] "RemoveContainer" containerID="6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8"
Oct 30 05:11:23.999658 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.999634    5072 scope.go:122] "RemoveContainer" containerID="6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8"
Oct 30 05:11:23.999947 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:11:23.999928    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8\": container with ID starting with 6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8 not found: ID does not exist" containerID="6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8"
Oct 30 05:11:24.000029 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:23.999956    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8"} err="failed to get container status \"6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8\": rpc error: code = NotFound desc = could not find container \"6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8\": container with ID starting with 6000cd6527f49aca9dbb52312c25b9b2990afde2bed955cb8eaed62a416897a8 not found: ID does not exist"
Oct 30 05:11:24.030309 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:24.030175    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv" podStartSLOduration=2.030162065 podStartE2EDuration="2.030162065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:24.028221044 +0000 UTC m=+4463.769644142" watchObservedRunningTime="2026-10-30 05:11:24.030162065 +0000 UTC m=+4463.771585152"
Oct 30 05:11:24.066506 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:24.066474    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"]
Oct 30 05:11:24.072389 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:24.072364    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/simulator-w1-6796b1-kserve-7f95bb55-f69mb"]
Oct 30 05:11:24.408925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:24.408896    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce161e2f-5b10-4d42-971e-0d51c68be921" path="/var/lib/kubelet/pods/ce161e2f-5b10-4d42-971e-0d51c68be921/volumes"
Oct 30 05:11:24.996783 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:24.996748    5072 generic.go:356] "Generic (PLEG): container finished" podID="2446496a-8472-4ded-b6f5-b4ef56f518b7" containerID="0e9c9bb13179ebc9703849e7819952774dc4ec503cd221c760325831a031638d" exitCode=0
Oct 30 05:11:24.997315 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:24.996833    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv" event={"ID":"2446496a-8472-4ded-b6f5-b4ef56f518b7","Type":"ContainerDied","Data":"0e9c9bb13179ebc9703849e7819952774dc4ec503cd221c760325831a031638d"}
Oct 30 05:11:26.164445 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.164409    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:26.295694 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.295607    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-openshift-ca\") pod \"2446496a-8472-4ded-b6f5-b4ef56f518b7\" (UID: \"2446496a-8472-4ded-b6f5-b4ef56f518b7\") "
Oct 30 05:11:26.295921 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.295896    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2446496a-8472-4ded-b6f5-b4ef56f518b7-kube-api-access-2rs45\" (UniqueName: \"kubernetes.io/projected/2446496a-8472-4ded-b6f5-b4ef56f518b7-kube-api-access-2rs45\") pod \"2446496a-8472-4ded-b6f5-b4ef56f518b7\" (UID: \"2446496a-8472-4ded-b6f5-b4ef56f518b7\") "
Oct 30 05:11:26.296002 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.295985    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-xks-ca\") pod \"2446496a-8472-4ded-b6f5-b4ef56f518b7\" (UID: \"2446496a-8472-4ded-b6f5-b4ef56f518b7\") "
Oct 30 05:11:26.296044 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.295996    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-openshift-ca" pod "2446496a-8472-4ded-b6f5-b4ef56f518b7" (UID: "2446496a-8472-4ded-b6f5-b4ef56f518b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:11:26.296495 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.296469    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:26.298197 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.298174    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-xks-ca" pod "2446496a-8472-4ded-b6f5-b4ef56f518b7" (UID: "2446496a-8472-4ded-b6f5-b4ef56f518b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:11:26.298262 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.298214    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2446496a-8472-4ded-b6f5-b4ef56f518b7-kube-api-access-2rs45" pod "2446496a-8472-4ded-b6f5-b4ef56f518b7" (UID: "2446496a-8472-4ded-b6f5-b4ef56f518b7"). InnerVolumeSpecName "kube-api-access-2rs45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:26.397497 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.397469    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2446496a-8472-4ded-b6f5-b4ef56f518b7-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:26.397497 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.397497    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rs45\" (UniqueName: \"kubernetes.io/projected/2446496a-8472-4ded-b6f5-b4ef56f518b7-kube-api-access-2rs45\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:26.981468 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.981433    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"]
Oct 30 05:11:26.982332 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.982303    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ce161e2f-5b10-4d42-971e-0d51c68be921" containerName="main"
Oct 30 05:11:26.982332 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.982325    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce161e2f-5b10-4d42-971e-0d51c68be921" containerName="main"
Oct 30 05:11:26.982512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.982341    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2446496a-8472-4ded-b6f5-b4ef56f518b7" containerName="revoke-keys"
Oct 30 05:11:26.982512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.982346    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2446496a-8472-4ded-b6f5-b4ef56f518b7" containerName="revoke-keys"
Oct 30 05:11:26.982512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.982471    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce161e2f-5b10-4d42-971e-0d51c68be921" containerName="main"
Oct 30 05:11:26.982512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.982480    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2446496a-8472-4ded-b6f5-b4ef56f518b7" containerName="revoke-keys"
Oct 30 05:11:26.986211 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:26.986192    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:27.002929 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.002905    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"]
Oct 30 05:11:27.018678 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.018620    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"
Oct 30 05:11:27.018678 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.018637    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv" event={"ID":"2446496a-8472-4ded-b6f5-b4ef56f518b7","Type":"ContainerDied","Data":"7a50cef3944cfb12999130a5023d8a65ec481be5e448fc8c6b2ab25d576596c8"}
Oct 30 05:11:27.018862 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.018690    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a50cef3944cfb12999130a5023d8a65ec481be5e448fc8c6b2ab25d576596c8"
Oct 30 05:11:27.106668 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.106633    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9f72815-9sd59\" (UID: \"f8e332a2-673b-4b12-94e7-48aec1d51716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:27.106863 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.106842    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9f72815-9sd59\" (UID: \"f8e332a2-673b-4b12-94e7-48aec1d51716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:27.106929 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.106918    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bcfj\" (UniqueName: \"kubernetes.io/projected/f8e332a2-673b-4b12-94e7-48aec1d51716-kube-api-access-9bcfj\") pod \"maas-api-revoke-sub-e9f72815-9sd59\" (UID: \"f8e332a2-673b-4b12-94e7-48aec1d51716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:27.207586 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.207554    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9f72815-9sd59\" (UID: \"f8e332a2-673b-4b12-94e7-48aec1d51716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:27.207964 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.207611    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bcfj\" (UniqueName: \"kubernetes.io/projected/f8e332a2-673b-4b12-94e7-48aec1d51716-kube-api-access-9bcfj\") pod \"maas-api-revoke-sub-e9f72815-9sd59\" (UID: \"f8e332a2-673b-4b12-94e7-48aec1d51716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:27.207964 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.207665    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9f72815-9sd59\" (UID: \"f8e332a2-673b-4b12-94e7-48aec1d51716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:27.208191 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.208162    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9f72815-9sd59\" (UID: \"f8e332a2-673b-4b12-94e7-48aec1d51716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:27.210368 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.210339    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9f72815-9sd59\" (UID: \"f8e332a2-673b-4b12-94e7-48aec1d51716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:27.231917 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.231863    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bcfj\" (UniqueName: \"kubernetes.io/projected/f8e332a2-673b-4b12-94e7-48aec1d51716-kube-api-access-9bcfj\") pod \"maas-api-revoke-sub-e9f72815-9sd59\" (UID: \"f8e332a2-673b-4b12-94e7-48aec1d51716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:27.300503 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.300479    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:27.613920 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:11:27.613887    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e332a2_673b_4b12_94e7_48aec1d51716.slice/crio-9b44354d2fe454196391905253043a3cf06a84b4638fb6af794614d8dd826346 WatchSource:0}: Error finding container 9b44354d2fe454196391905253043a3cf06a84b4638fb6af794614d8dd826346: Status 404 returned error can't find the container with id 9b44354d2fe454196391905253043a3cf06a84b4638fb6af794614d8dd826346
Oct 30 05:11:27.615649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:27.615631    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"]
Oct 30 05:11:28.028833 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:28.028801    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59" event={"ID":"f8e332a2-673b-4b12-94e7-48aec1d51716","Type":"ContainerStarted","Data":"423bac4f817e00192174a32b31dadab71d29bfb962f96cf4c8f22dc0607d68a3"}
Oct 30 05:11:28.029009 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:28.028854    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59" event={"ID":"f8e332a2-673b-4b12-94e7-48aec1d51716","Type":"ContainerStarted","Data":"9b44354d2fe454196391905253043a3cf06a84b4638fb6af794614d8dd826346"}
Oct 30 05:11:28.075206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:28.075154    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59" podStartSLOduration=2.075139793 podStartE2EDuration="2.075139793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:28.072649985 +0000 UTC m=+4467.814073079" watchObservedRunningTime="2026-10-30 05:11:28.075139793 +0000 UTC m=+4467.816562892"
Oct 30 05:11:29.036860 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:29.036826    5072 generic.go:356] "Generic (PLEG): container finished" podID="f8e332a2-673b-4b12-94e7-48aec1d51716" containerID="423bac4f817e00192174a32b31dadab71d29bfb962f96cf4c8f22dc0607d68a3" exitCode=0
Oct 30 05:11:29.037299 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:29.036922    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59" event={"ID":"f8e332a2-673b-4b12-94e7-48aec1d51716","Type":"ContainerDied","Data":"423bac4f817e00192174a32b31dadab71d29bfb962f96cf4c8f22dc0607d68a3"}
Oct 30 05:11:30.200535 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:30.200510    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:30.345580 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:30.345509    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-xks-ca\") pod \"f8e332a2-673b-4b12-94e7-48aec1d51716\" (UID: \"f8e332a2-673b-4b12-94e7-48aec1d51716\") "
Oct 30 05:11:30.345720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:30.345642    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-openshift-ca\") pod \"f8e332a2-673b-4b12-94e7-48aec1d51716\" (UID: \"f8e332a2-673b-4b12-94e7-48aec1d51716\") "
Oct 30 05:11:30.345759 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:30.345728    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e332a2-673b-4b12-94e7-48aec1d51716-kube-api-access-9bcfj\" (UniqueName: \"kubernetes.io/projected/f8e332a2-673b-4b12-94e7-48aec1d51716-kube-api-access-9bcfj\") pod \"f8e332a2-673b-4b12-94e7-48aec1d51716\" (UID: \"f8e332a2-673b-4b12-94e7-48aec1d51716\") "
Oct 30 05:11:30.346005 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:30.345980    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-openshift-ca" pod "f8e332a2-673b-4b12-94e7-48aec1d51716" (UID: "f8e332a2-673b-4b12-94e7-48aec1d51716"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:11:30.346266 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:30.346137    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:30.347887 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:30.347856    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-xks-ca" pod "f8e332a2-673b-4b12-94e7-48aec1d51716" (UID: "f8e332a2-673b-4b12-94e7-48aec1d51716"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:11:30.348036 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:30.348010    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e332a2-673b-4b12-94e7-48aec1d51716-kube-api-access-9bcfj" pod "f8e332a2-673b-4b12-94e7-48aec1d51716" (UID: "f8e332a2-673b-4b12-94e7-48aec1d51716"). InnerVolumeSpecName "kube-api-access-9bcfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:30.447702 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:30.447664    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e332a2-673b-4b12-94e7-48aec1d51716-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:30.447702 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:30.447703    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bcfj\" (UniqueName: \"kubernetes.io/projected/f8e332a2-673b-4b12-94e7-48aec1d51716-kube-api-access-9bcfj\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:11:31.052721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:31.052693    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"
Oct 30 05:11:31.052922 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:31.052680    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59" event={"ID":"f8e332a2-673b-4b12-94e7-48aec1d51716","Type":"ContainerDied","Data":"9b44354d2fe454196391905253043a3cf06a84b4638fb6af794614d8dd826346"}
Oct 30 05:11:31.052922 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:31.052835    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b44354d2fe454196391905253043a3cf06a84b4638fb6af794614d8dd826346"
Oct 30 05:11:41.752843 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:11:41.752797    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:11:45.484511 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:45.484480    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"]
Oct 30 05:11:58.727585 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:11:58.727547    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"]
Oct 30 05:12:00.537127 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.537054    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m" podUID="79eed7f5-2953-4479-9b27-0eefd7223518" containerName="main" containerID="cri-o://6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d" gracePeriod=45
Oct 30 05:12:00.823437 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.823407    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:12:00.941277 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.941218    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-home\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-home\") pod \"79eed7f5-2953-4479-9b27-0eefd7223518\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") "
Oct 30 05:12:00.941557 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.941289    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-dshm\") pod \"79eed7f5-2953-4479-9b27-0eefd7223518\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") "
Oct 30 05:12:00.941557 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.941321    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-model-cache\") pod \"79eed7f5-2953-4479-9b27-0eefd7223518\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") "
Oct 30 05:12:00.941557 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.941374    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79eed7f5-2953-4479-9b27-0eefd7223518-tls-certs\" (UniqueName: \"kubernetes.io/secret/79eed7f5-2953-4479-9b27-0eefd7223518-tls-certs\") pod \"79eed7f5-2953-4479-9b27-0eefd7223518\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") "
Oct 30 05:12:00.942288 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.941587    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79eed7f5-2953-4479-9b27-0eefd7223518-kube-api-access-g98cp\" (UniqueName: \"kubernetes.io/projected/79eed7f5-2953-4479-9b27-0eefd7223518-kube-api-access-g98cp\") pod \"79eed7f5-2953-4479-9b27-0eefd7223518\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") "
Oct 30 05:12:00.942288 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.941660    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-tmp-dir\") pod \"79eed7f5-2953-4479-9b27-0eefd7223518\" (UID: \"79eed7f5-2953-4479-9b27-0eefd7223518\") "
Oct 30 05:12:00.942838 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.942613    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-tmp-dir" pod "79eed7f5-2953-4479-9b27-0eefd7223518" (UID: "79eed7f5-2953-4479-9b27-0eefd7223518"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:00.942838 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.942801    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-model-cache" pod "79eed7f5-2953-4479-9b27-0eefd7223518" (UID: "79eed7f5-2953-4479-9b27-0eefd7223518"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:00.943007 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.942839    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-home" pod "79eed7f5-2953-4479-9b27-0eefd7223518" (UID: "79eed7f5-2953-4479-9b27-0eefd7223518"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:00.945195 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.945157    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79eed7f5-2953-4479-9b27-0eefd7223518-tls-certs" pod "79eed7f5-2953-4479-9b27-0eefd7223518" (UID: "79eed7f5-2953-4479-9b27-0eefd7223518"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:12:00.946496 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.946464    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-dshm" pod "79eed7f5-2953-4479-9b27-0eefd7223518" (UID: "79eed7f5-2953-4479-9b27-0eefd7223518"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:00.946745 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:00.946719    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79eed7f5-2953-4479-9b27-0eefd7223518-kube-api-access-g98cp" pod "79eed7f5-2953-4479-9b27-0eefd7223518" (UID: "79eed7f5-2953-4479-9b27-0eefd7223518"). InnerVolumeSpecName "kube-api-access-g98cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:01.044116 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.044005    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:01.044116 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.044034    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:01.044116 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.044046    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79eed7f5-2953-4479-9b27-0eefd7223518-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:01.044116 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.044055    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g98cp\" (UniqueName: \"kubernetes.io/projected/79eed7f5-2953-4479-9b27-0eefd7223518-kube-api-access-g98cp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:01.044116 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.044064    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:01.044116 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.044072    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79eed7f5-2953-4479-9b27-0eefd7223518-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:01.113715 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.113683    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:12:01.136498 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.136472    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:12:01.283971 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.283939    5072 generic.go:356] "Generic (PLEG): container finished" podID="79eed7f5-2953-4479-9b27-0eefd7223518" containerID="6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d" exitCode=0
Oct 30 05:12:01.284222 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.283989    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"
Oct 30 05:12:01.284222 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.284033    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m" event={"ID":"79eed7f5-2953-4479-9b27-0eefd7223518","Type":"ContainerDied","Data":"6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d"}
Oct 30 05:12:01.284222 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.284097    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m" event={"ID":"79eed7f5-2953-4479-9b27-0eefd7223518","Type":"ContainerDied","Data":"d59f08cd65ab5360cb924a3c0c2b9be53352534ebea52ba7cb389192fbf2e044"}
Oct 30 05:12:01.284222 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.284121    5072 scope.go:122] "RemoveContainer" containerID="6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d"
Oct 30 05:12:01.295534 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.295517    5072 scope.go:122] "RemoveContainer" containerID="6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d"
Oct 30 05:12:01.295794 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:01.295772    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d\": container with ID starting with 6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d not found: ID does not exist" containerID="6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d"
Oct 30 05:12:01.295875 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.295802    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d"} err="failed to get container status \"6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d\": rpc error: code = NotFound desc = could not find container \"6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d\": container with ID starting with 6889582d04e4dae3453006d5ea01ddc238c0f36b9c5055953bbdc9d4e758986d not found: ID does not exist"
Oct 30 05:12:01.357579 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.357549    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"]
Oct 30 05:12:01.370666 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:01.370627    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4c2415/rate-test-model-4c2415-kserve-5b768b997c-hck8m"]
Oct 30 05:12:02.410934 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:02.410898    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79eed7f5-2953-4479-9b27-0eefd7223518" path="/var/lib/kubelet/pods/79eed7f5-2953-4479-9b27-0eefd7223518/volumes"
Oct 30 05:12:03.309857 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:03.309824    5072 scope.go:122] "RemoveContainer" containerID="ff9508980ea7dc66fa78c4343fc472d5a382382cd015c3a6c3065d4f7bf94a90"
Oct 30 05:12:03.320322 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:03.320303    5072 scope.go:122] "RemoveContainer" containerID="e03c6880e648676f8510235796ee66ac1ad579b02f831c13baf3af4bc0827ac1"
Oct 30 05:12:03.329597 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:03.329579    5072 scope.go:122] "RemoveContainer" containerID="d71669cfaf70b96d50bc6953fffce1569de269078e07eb8b84f0bc627d12a7c0"
Oct 30 05:12:03.338746 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:03.338722    5072 scope.go:122] "RemoveContainer" containerID="1fcff1b5e15fc389ab7b97252cdd9d223c5bc33ca4003f37d9ab4be3398e6133"
Oct 30 05:12:06.295441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:06.295392    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"]
Oct 30 05:12:06.296695 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:06.296652    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb" podUID="5c1b7028-ef70-47a1-a530-275f970c88aa" containerName="istio-proxy" containerID="cri-o://e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600" gracePeriod=30
Oct 30 05:12:11.593538 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.593469    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 05:12:11.678277 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.678251    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-token\" (UniqueName: \"kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-token\") pod \"5c1b7028-ef70-47a1-a530-275f970c88aa\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") "
Oct 30 05:12:11.678459 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.678327    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-envoy\") pod \"5c1b7028-ef70-47a1-a530-275f970c88aa\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") "
Oct 30 05:12:11.678459 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.678387    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-certs\") pod \"5c1b7028-ef70-47a1-a530-275f970c88aa\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") "
Oct 30 05:12:11.678459 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.678423    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-credential-socket\") pod \"5c1b7028-ef70-47a1-a530-275f970c88aa\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") "
Oct 30 05:12:11.678459 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.678456    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-socket\") pod \"5c1b7028-ef70-47a1-a530-275f970c88aa\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") "
Oct 30 05:12:11.678729 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.678578    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-kube-api-access-frrw4\" (UniqueName: \"kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-kube-api-access-frrw4\") pod \"5c1b7028-ef70-47a1-a530-275f970c88aa\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") "
Oct 30 05:12:11.678729 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.678638    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-data\") pod \"5c1b7028-ef70-47a1-a530-275f970c88aa\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") "
Oct 30 05:12:11.678879 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.678856    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-data" pod "5c1b7028-ef70-47a1-a530-275f970c88aa" (UID: "5c1b7028-ef70-47a1-a530-275f970c88aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:11.678948 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.678929    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-podinfo\") pod \"5c1b7028-ef70-47a1-a530-275f970c88aa\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") "
Oct 30 05:12:11.679095 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.678964    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-credential-socket" pod "5c1b7028-ef70-47a1-a530-275f970c88aa" (UID: "5c1b7028-ef70-47a1-a530-275f970c88aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:11.679095 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.678986    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c1b7028-ef70-47a1-a530-275f970c88aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1b7028-ef70-47a1-a530-275f970c88aa-istiod-ca-cert\") pod \"5c1b7028-ef70-47a1-a530-275f970c88aa\" (UID: \"5c1b7028-ef70-47a1-a530-275f970c88aa\") "
Oct 30 05:12:11.679385 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.679257    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-certs" pod "5c1b7028-ef70-47a1-a530-275f970c88aa" (UID: "5c1b7028-ef70-47a1-a530-275f970c88aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:11.679385 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.679283    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-socket" pod "5c1b7028-ef70-47a1-a530-275f970c88aa" (UID: "5c1b7028-ef70-47a1-a530-275f970c88aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:11.679599 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.679575    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1b7028-ef70-47a1-a530-275f970c88aa-istiod-ca-cert" pod "5c1b7028-ef70-47a1-a530-275f970c88aa" (UID: "5c1b7028-ef70-47a1-a530-275f970c88aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:12:11.679873 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.679804    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:11.679873 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.679840    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:11.679873 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.679857    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:11.679873 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.679873    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:11.680929 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.680906    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-envoy" pod "5c1b7028-ef70-47a1-a530-275f970c88aa" (UID: "5c1b7028-ef70-47a1-a530-275f970c88aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:11.681423 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.681393    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-podinfo" pod "5c1b7028-ef70-47a1-a530-275f970c88aa" (UID: "5c1b7028-ef70-47a1-a530-275f970c88aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:12:11.681423 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.681391    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-token" pod "5c1b7028-ef70-47a1-a530-275f970c88aa" (UID: "5c1b7028-ef70-47a1-a530-275f970c88aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:11.681803 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.681786    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-kube-api-access-frrw4" pod "5c1b7028-ef70-47a1-a530-275f970c88aa" (UID: "5c1b7028-ef70-47a1-a530-275f970c88aa"). InnerVolumeSpecName "kube-api-access-frrw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:11.766594 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:11.766554    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:12:11.781510 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.781481    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frrw4\" (UniqueName: \"kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-kube-api-access-frrw4\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:11.781510 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.781511    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:11.781673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.781523    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1b7028-ef70-47a1-a530-275f970c88aa-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:11.781673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.781532    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:11.781673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:11.781541    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1b7028-ef70-47a1-a530-275f970c88aa-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:12.366931 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:12.366901    5072 generic.go:356] "Generic (PLEG): container finished" podID="5c1b7028-ef70-47a1-a530-275f970c88aa" containerID="e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600" exitCode=0
Oct 30 05:12:12.367153 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:12.366951    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"
Oct 30 05:12:12.367153 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:12.366994    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb" event={"ID":"5c1b7028-ef70-47a1-a530-275f970c88aa","Type":"ContainerDied","Data":"e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600"}
Oct 30 05:12:12.367153 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:12.367048    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb" event={"ID":"5c1b7028-ef70-47a1-a530-275f970c88aa","Type":"ContainerDied","Data":"53468e846c72a58a82dfc5fbf599ff262c478c4b64c802e4bae78a3750ad8624"}
Oct 30 05:12:12.367153 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:12.367109    5072 scope.go:122] "RemoveContainer" containerID="e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600"
Oct 30 05:12:12.378591 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:12.378570    5072 scope.go:122] "RemoveContainer" containerID="e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600"
Oct 30 05:12:12.378874 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:12.378856    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600\": container with ID starting with e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600 not found: ID does not exist" containerID="e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600"
Oct 30 05:12:12.378935 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:12.378882    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600"} err="failed to get container status \"e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600\": rpc error: code = NotFound desc = could not find container \"e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600\": container with ID starting with e0269f3310a1ad522cb87af44849c2d449606a3fcf4eaf4c181cc078277a5600 not found: ID does not exist"
Oct 30 05:12:12.434887 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:12.434853    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"]
Oct 30 05:12:12.447598 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:12.447571    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-76qzb"]
Oct 30 05:12:13.763463 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:13.763421    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk" podUID="f046af00-47a2-4493-8b22-3b3bdc2b0d02" containerName="main" containerID="cri-o://e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2" gracePeriod=45
Oct 30 05:12:14.055845 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.055815    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:12:14.108339 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.108309    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f046af00-47a2-4493-8b22-3b3bdc2b0d02-kube-api-access-h6sxw\" (UniqueName: \"kubernetes.io/projected/f046af00-47a2-4493-8b22-3b3bdc2b0d02-kube-api-access-h6sxw\") pod \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") "
Oct 30 05:12:14.108529 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.108363    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-dshm\") pod \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") "
Oct 30 05:12:14.108529 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.108472    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tls-certs\" (UniqueName: \"kubernetes.io/secret/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tls-certs\") pod \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") "
Oct 30 05:12:14.108662 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.108619    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tmp-dir\") pod \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") "
Oct 30 05:12:14.108759 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.108744    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-model-cache\") pod \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") "
Oct 30 05:12:14.108821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.108781    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-home\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-home\") pod \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\" (UID: \"f046af00-47a2-4493-8b22-3b3bdc2b0d02\") "
Oct 30 05:12:14.109059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.109030    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tmp-dir" pod "f046af00-47a2-4493-8b22-3b3bdc2b0d02" (UID: "f046af00-47a2-4493-8b22-3b3bdc2b0d02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:14.109201 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.109059    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-home" pod "f046af00-47a2-4493-8b22-3b3bdc2b0d02" (UID: "f046af00-47a2-4493-8b22-3b3bdc2b0d02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:14.109273 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.109229    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-model-cache" pod "f046af00-47a2-4493-8b22-3b3bdc2b0d02" (UID: "f046af00-47a2-4493-8b22-3b3bdc2b0d02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:14.109383 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.109361    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:14.109456 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.109406    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:14.111002 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.110979    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-dshm" pod "f046af00-47a2-4493-8b22-3b3bdc2b0d02" (UID: "f046af00-47a2-4493-8b22-3b3bdc2b0d02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:14.111756 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.111733    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tls-certs" pod "f046af00-47a2-4493-8b22-3b3bdc2b0d02" (UID: "f046af00-47a2-4493-8b22-3b3bdc2b0d02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:12:14.111843 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.111786    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f046af00-47a2-4493-8b22-3b3bdc2b0d02-kube-api-access-h6sxw" pod "f046af00-47a2-4493-8b22-3b3bdc2b0d02" (UID: "f046af00-47a2-4493-8b22-3b3bdc2b0d02"). InnerVolumeSpecName "kube-api-access-h6sxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:14.211002 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.210964    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6sxw\" (UniqueName: \"kubernetes.io/projected/f046af00-47a2-4493-8b22-3b3bdc2b0d02-kube-api-access-h6sxw\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:14.211002 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.211001    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:14.211232 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.211011    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f046af00-47a2-4493-8b22-3b3bdc2b0d02-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:14.211232 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.211021    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f046af00-47a2-4493-8b22-3b3bdc2b0d02-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:14.385362 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.385278    5072 generic.go:356] "Generic (PLEG): container finished" podID="f046af00-47a2-4493-8b22-3b3bdc2b0d02" containerID="e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2" exitCode=0
Oct 30 05:12:14.385362 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.385306    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk" event={"ID":"f046af00-47a2-4493-8b22-3b3bdc2b0d02","Type":"ContainerDied","Data":"e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2"}
Oct 30 05:12:14.385362 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.385340    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"
Oct 30 05:12:14.385362 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.385368    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk" event={"ID":"f046af00-47a2-4493-8b22-3b3bdc2b0d02","Type":"ContainerDied","Data":"a7e5ff9fb6797fb6a2ecef8e2c8430e83e0b57ad33217fa765180e9611d8ff77"}
Oct 30 05:12:14.385725 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.385393    5072 scope.go:122] "RemoveContainer" containerID="e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2"
Oct 30 05:12:14.397056 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.397039    5072 scope.go:122] "RemoveContainer" containerID="e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2"
Oct 30 05:12:14.397358 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:14.397332    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2\": container with ID starting with e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2 not found: ID does not exist" containerID="e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2"
Oct 30 05:12:14.397443 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.397367    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2"} err="failed to get container status \"e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2\": rpc error: code = NotFound desc = could not find container \"e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2\": container with ID starting with e92179775933a957502bf4a5d22a74bc809fe7c3ddd3635ac9d3afc8fa5663b2 not found: ID does not exist"
Oct 30 05:12:14.409163 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.409138    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1b7028-ef70-47a1-a530-275f970c88aa" path="/var/lib/kubelet/pods/5c1b7028-ef70-47a1-a530-275f970c88aa/volumes"
Oct 30 05:12:14.442340 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.442305    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"]
Oct 30 05:12:14.454770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.454740    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-bbce72/rate-test-model-bbce72-kserve-7d9f8854b5-rn2qk"]
Oct 30 05:12:14.457992 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.457969    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"]
Oct 30 05:12:14.458207 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:14.458184    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy" containerID="cri-o://5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f" gracePeriod=30
Oct 30 05:12:16.408808 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:16.408773    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f046af00-47a2-4493-8b22-3b3bdc2b0d02" path="/var/lib/kubelet/pods/f046af00-47a2-4493-8b22-3b3bdc2b0d02/volumes"
Oct 30 05:12:19.731195 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.731136    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 05:12:19.880645 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.880617    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-istio-token\" (UniqueName: \"kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-istio-token\") pod \"04291e68-092a-4c76-bb56-52ce38990484\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") "
Oct 30 05:12:19.880807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.880670    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-kube-api-access-c8vpb\" (UniqueName: \"kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-kube-api-access-c8vpb\") pod \"04291e68-092a-4c76-bb56-52ce38990484\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") "
Oct 30 05:12:19.880807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.880692    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04291e68-092a-4c76-bb56-52ce38990484-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04291e68-092a-4c76-bb56-52ce38990484-istio-podinfo\") pod \"04291e68-092a-4c76-bb56-52ce38990484\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") "
Oct 30 05:12:19.880807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.880714    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-data\") pod \"04291e68-092a-4c76-bb56-52ce38990484\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") "
Oct 30 05:12:19.880807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.880747    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-envoy\") pod \"04291e68-092a-4c76-bb56-52ce38990484\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") "
Oct 30 05:12:19.880807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.880778    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-socket\") pod \"04291e68-092a-4c76-bb56-52ce38990484\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") "
Oct 30 05:12:19.881110 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.880919    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04291e68-092a-4c76-bb56-52ce38990484-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04291e68-092a-4c76-bb56-52ce38990484-istiod-ca-cert\") pod \"04291e68-092a-4c76-bb56-52ce38990484\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") "
Oct 30 05:12:19.881110 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.880956    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-credential-socket\") pod \"04291e68-092a-4c76-bb56-52ce38990484\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") "
Oct 30 05:12:19.881110 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.881001    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-certs\") pod \"04291e68-092a-4c76-bb56-52ce38990484\" (UID: \"04291e68-092a-4c76-bb56-52ce38990484\") "
Oct 30 05:12:19.881304 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.881149    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-data" pod "04291e68-092a-4c76-bb56-52ce38990484" (UID: "04291e68-092a-4c76-bb56-52ce38990484"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:19.881458 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.881432    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-credential-socket" pod "04291e68-092a-4c76-bb56-52ce38990484" (UID: "04291e68-092a-4c76-bb56-52ce38990484"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:19.881559 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.881516    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:19.881559 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.881534    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:19.881850 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.881660    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-certs" pod "04291e68-092a-4c76-bb56-52ce38990484" (UID: "04291e68-092a-4c76-bb56-52ce38990484"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:19.881850 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.881709    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-socket" pod "04291e68-092a-4c76-bb56-52ce38990484" (UID: "04291e68-092a-4c76-bb56-52ce38990484"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:19.881972 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.881878    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04291e68-092a-4c76-bb56-52ce38990484-istiod-ca-cert" pod "04291e68-092a-4c76-bb56-52ce38990484" (UID: "04291e68-092a-4c76-bb56-52ce38990484"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:12:19.883292 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.883264    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04291e68-092a-4c76-bb56-52ce38990484-istio-podinfo" pod "04291e68-092a-4c76-bb56-52ce38990484" (UID: "04291e68-092a-4c76-bb56-52ce38990484"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:12:19.883292 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.883277    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-envoy" pod "04291e68-092a-4c76-bb56-52ce38990484" (UID: "04291e68-092a-4c76-bb56-52ce38990484"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:19.883522 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.883502    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-istio-token" pod "04291e68-092a-4c76-bb56-52ce38990484" (UID: "04291e68-092a-4c76-bb56-52ce38990484"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:19.883582 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.883546    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-kube-api-access-c8vpb" pod "04291e68-092a-4c76-bb56-52ce38990484" (UID: "04291e68-092a-4c76-bb56-52ce38990484"). InnerVolumeSpecName "kube-api-access-c8vpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:19.982807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.982734    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:19.982807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.982761    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vpb\" (UniqueName: \"kubernetes.io/projected/04291e68-092a-4c76-bb56-52ce38990484-kube-api-access-c8vpb\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:19.982807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.982771    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04291e68-092a-4c76-bb56-52ce38990484-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:19.982807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.982780    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:19.982807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.982788    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:19.982807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.982797    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04291e68-092a-4c76-bb56-52ce38990484-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:19.982807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:19.982806    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04291e68-092a-4c76-bb56-52ce38990484-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:20.427651 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:20.427611    5072 generic.go:356] "Generic (PLEG): container finished" podID="04291e68-092a-4c76-bb56-52ce38990484" containerID="5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f" exitCode=0
Oct 30 05:12:20.427807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:20.427667    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" event={"ID":"04291e68-092a-4c76-bb56-52ce38990484","Type":"ContainerDied","Data":"5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f"}
Oct 30 05:12:20.427807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:20.427679    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"
Oct 30 05:12:20.427807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:20.427707    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm" event={"ID":"04291e68-092a-4c76-bb56-52ce38990484","Type":"ContainerDied","Data":"1c9c8ca862f157ba55c926b3635f7bf3838ac3c83c26daa770ae371bf7ab984d"}
Oct 30 05:12:20.427807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:20.427728    5072 scope.go:122] "RemoveContainer" containerID="5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f"
Oct 30 05:12:20.438579 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:20.438559    5072 scope.go:122] "RemoveContainer" containerID="cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec"
Oct 30 05:12:20.448179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:20.448146    5072 scope.go:122] "RemoveContainer" containerID="5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f"
Oct 30 05:12:20.448385 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:20.448369    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f\": container with ID starting with 5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f not found: ID does not exist" containerID="5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f"
Oct 30 05:12:20.448440 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:20.448391    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f"} err="failed to get container status \"5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f\": rpc error: code = NotFound desc = could not find container \"5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f\": container with ID starting with 5fbd7e799d805c4b1c74a8ac804c5a6c90a2b73534699bda868b7a83ce6e209f not found: ID does not exist"
Oct 30 05:12:20.448440 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:20.448407    5072 scope.go:122] "RemoveContainer" containerID="cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec"
Oct 30 05:12:20.448656 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:20.448643    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec\": container with ID starting with cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec not found: ID does not exist" containerID="cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec"
Oct 30 05:12:20.448702 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:20.448659    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec"} err="failed to get container status \"cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec\": rpc error: code = NotFound desc = could not find container \"cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec\": container with ID starting with cbe86ed7965b4344b383d1a385276884e9752012b595e735356fa027756ea3ec not found: ID does not exist"
Oct 30 05:12:20.484093 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:20.484055    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"]
Oct 30 05:12:20.496805 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:20.496775    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0d6e7-openshift-default-5d4d6c5549-sgwqm"]
Oct 30 05:12:22.408251 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:22.408221    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04291e68-092a-4c76-bb56-52ce38990484" path="/var/lib/kubelet/pods/04291e68-092a-4c76-bb56-52ce38990484/volumes"
Oct 30 05:12:33.039820 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:33.039781    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"]
Oct 30 05:12:33.052672 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:33.052640    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b971c5-bcthc"]
Oct 30 05:12:34.419435 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:34.419397    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0674b215-1609-416c-8921-13b00c6ad543" path="/var/lib/kubelet/pods/0674b215-1609-416c-8921-13b00c6ad543/volumes"
Oct 30 05:12:38.061897 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.061855    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11252-5d15eb"]
Oct 30 05:12:38.062751 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.062725    5072 cpu_manager.go:414] "containerMap: removing container" podUID="5c1b7028-ef70-47a1-a530-275f970c88aa" containerName="istio-proxy"
Oct 30 05:12:38.062859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.062754    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1b7028-ef70-47a1-a530-275f970c88aa" containerName="istio-proxy"
Oct 30 05:12:38.062859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.062768    5072 cpu_manager.go:414] "containerMap: removing container" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy"
Oct 30 05:12:38.062859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.062776    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy"
Oct 30 05:12:38.062859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.062792    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f046af00-47a2-4493-8b22-3b3bdc2b0d02" containerName="main"
Oct 30 05:12:38.062859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.062799    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f046af00-47a2-4493-8b22-3b3bdc2b0d02" containerName="main"
Oct 30 05:12:38.062859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.062816    5072 cpu_manager.go:414] "containerMap: removing container" podUID="79eed7f5-2953-4479-9b27-0eefd7223518" containerName="main"
Oct 30 05:12:38.062859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.062823    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79eed7f5-2953-4479-9b27-0eefd7223518" containerName="main"
Oct 30 05:12:38.062859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.062858    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f8e332a2-673b-4b12-94e7-48aec1d51716" containerName="revoke-keys"
Oct 30 05:12:38.063319 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.062866    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e332a2-673b-4b12-94e7-48aec1d51716" containerName="revoke-keys"
Oct 30 05:12:38.063319 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.063035    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1b7028-ef70-47a1-a530-275f970c88aa" containerName="istio-proxy"
Oct 30 05:12:38.063319 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.063047    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy"
Oct 30 05:12:38.063319 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.063057    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f046af00-47a2-4493-8b22-3b3bdc2b0d02" containerName="main"
Oct 30 05:12:38.063319 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.063069    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="79eed7f5-2953-4479-9b27-0eefd7223518" containerName="main"
Oct 30 05:12:38.063319 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.063102    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e332a2-673b-4b12-94e7-48aec1d51716" containerName="revoke-keys"
Oct 30 05:12:38.063319 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.063113    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy"
Oct 30 05:12:38.068356 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.068327    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11252-5d15eb"
Oct 30 05:12:38.081432 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.081406    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11252-5d15eb"]
Oct 30 05:12:38.089965 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.089944    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gsgvm\""
Oct 30 05:12:38.112359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.112325    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlb4d\" (UniqueName: \"kubernetes.io/projected/7e383c67-de6f-4b3b-abc5-54f6aa6d43ed-kube-api-access-xlb4d\") pod \"test-curl-11252-5d15eb\" (UID: \"7e383c67-de6f-4b3b-abc5-54f6aa6d43ed\") " pod="opendatahub/test-curl-11252-5d15eb"
Oct 30 05:12:38.213717 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.213686    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlb4d\" (UniqueName: \"kubernetes.io/projected/7e383c67-de6f-4b3b-abc5-54f6aa6d43ed-kube-api-access-xlb4d\") pod \"test-curl-11252-5d15eb\" (UID: \"7e383c67-de6f-4b3b-abc5-54f6aa6d43ed\") " pod="opendatahub/test-curl-11252-5d15eb"
Oct 30 05:12:38.237744 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.237713    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlb4d\" (UniqueName: \"kubernetes.io/projected/7e383c67-de6f-4b3b-abc5-54f6aa6d43ed-kube-api-access-xlb4d\") pod \"test-curl-11252-5d15eb\" (UID: \"7e383c67-de6f-4b3b-abc5-54f6aa6d43ed\") " pod="opendatahub/test-curl-11252-5d15eb"
Oct 30 05:12:38.382421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.382341    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11252-5d15eb"
Oct 30 05:12:38.685548 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:12:38.685518    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e383c67_de6f_4b3b_abc5_54f6aa6d43ed.slice/crio-b8966acf58783a7d90923b3901c2436c34a67a75fada63bb264e1ae06535309c WatchSource:0}: Error finding container b8966acf58783a7d90923b3901c2436c34a67a75fada63bb264e1ae06535309c: Status 404 returned error can't find the container with id b8966acf58783a7d90923b3901c2436c34a67a75fada63bb264e1ae06535309c
Oct 30 05:12:38.687464 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:38.687436    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11252-5d15eb"]
Oct 30 05:12:39.570955 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:39.570917    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11252-5d15eb" event={"ID":"7e383c67-de6f-4b3b-abc5-54f6aa6d43ed","Type":"ContainerStarted","Data":"31f553a9d18cc3c876a85d325f0556eb1a04783609296a37e26cd1f4c902e050"}
Oct 30 05:12:39.570955 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:39.570955    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11252-5d15eb" event={"ID":"7e383c67-de6f-4b3b-abc5-54f6aa6d43ed","Type":"ContainerStarted","Data":"b8966acf58783a7d90923b3901c2436c34a67a75fada63bb264e1ae06535309c"}
Oct 30 05:12:39.609875 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:39.609831    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11252-5d15eb" podStartSLOduration=1.609819431 podStartE2EDuration="1.609819431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:39.609351443 +0000 UTC m=+4539.350774540" watchObservedRunningTime="2026-10-30 05:12:39.609819431 +0000 UTC m=+4539.351242527"
Oct 30 05:12:40.453432 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.453371    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11252-5d15eb"]
Oct 30 05:12:40.458486 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.458456    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11252-5d15eb"]
Oct 30 05:12:40.615832 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.615797    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11252-aef277"]
Oct 30 05:12:40.616833 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.616810    5072 cpu_manager.go:414] "containerMap: removing container" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy"
Oct 30 05:12:40.616883 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.616837    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04291e68-092a-4c76-bb56-52ce38990484" containerName="istio-proxy"
Oct 30 05:12:40.616883 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.616865    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7e383c67-de6f-4b3b-abc5-54f6aa6d43ed" containerName="test-curl-11252-5d15eb"
Oct 30 05:12:40.616883 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.616873    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e383c67-de6f-4b3b-abc5-54f6aa6d43ed" containerName="test-curl-11252-5d15eb"
Oct 30 05:12:40.617052 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.617039    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e383c67-de6f-4b3b-abc5-54f6aa6d43ed" containerName="test-curl-11252-5d15eb"
Oct 30 05:12:40.620803 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.620785    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11252-aef277"
Oct 30 05:12:40.637230 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.637200    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11252-aef277"]
Oct 30 05:12:40.642331 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.642308    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qgrk\" (UniqueName: \"kubernetes.io/projected/aecb42b8-bb7f-4d92-b894-ecb176176c0c-kube-api-access-6qgrk\") pod \"test-curl-11252-aef277\" (UID: \"aecb42b8-bb7f-4d92-b894-ecb176176c0c\") " pod="opendatahub/test-curl-11252-aef277"
Oct 30 05:12:40.743601 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.743509    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qgrk\" (UniqueName: \"kubernetes.io/projected/aecb42b8-bb7f-4d92-b894-ecb176176c0c-kube-api-access-6qgrk\") pod \"test-curl-11252-aef277\" (UID: \"aecb42b8-bb7f-4d92-b894-ecb176176c0c\") " pod="opendatahub/test-curl-11252-aef277"
Oct 30 05:12:40.767251 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.767218    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qgrk\" (UniqueName: \"kubernetes.io/projected/aecb42b8-bb7f-4d92-b894-ecb176176c0c-kube-api-access-6qgrk\") pod \"test-curl-11252-aef277\" (UID: \"aecb42b8-bb7f-4d92-b894-ecb176176c0c\") " pod="opendatahub/test-curl-11252-aef277"
Oct 30 05:12:40.935199 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:40.935167    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11252-aef277"
Oct 30 05:12:41.248933 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:41.248879    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11252-aef277"]
Oct 30 05:12:41.249112 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:12:41.249052    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaecb42b8_bb7f_4d92_b894_ecb176176c0c.slice/crio-2f24217606c5c53a50e119a7b9a59d0c10ad00eac0f1066f6d5f17ba5b51a9c3 WatchSource:0}: Error finding container 2f24217606c5c53a50e119a7b9a59d0c10ad00eac0f1066f6d5f17ba5b51a9c3: Status 404 returned error can't find the container with id 2f24217606c5c53a50e119a7b9a59d0c10ad00eac0f1066f6d5f17ba5b51a9c3
Oct 30 05:12:41.585729 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:41.585634    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11252-aef277" event={"ID":"aecb42b8-bb7f-4d92-b894-ecb176176c0c","Type":"ContainerStarted","Data":"5d50fc59113a4c4e5dd7c136b4707345cd0662f516ebd4f39bec5f224935ff73"}
Oct 30 05:12:41.585729 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:41.585695    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11252-aef277" event={"ID":"aecb42b8-bb7f-4d92-b894-ecb176176c0c","Type":"ContainerStarted","Data":"2f24217606c5c53a50e119a7b9a59d0c10ad00eac0f1066f6d5f17ba5b51a9c3"}
Oct 30 05:12:41.585965 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:41.585736    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11252-5d15eb" podUID="7e383c67-de6f-4b3b-abc5-54f6aa6d43ed" containerName="test-curl-11252-5d15eb" containerID="cri-o://31f553a9d18cc3c876a85d325f0556eb1a04783609296a37e26cd1f4c902e050" gracePeriod=2
Oct 30 05:12:41.600003 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:41.599971    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11252-5d15eb\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="7e383c67-de6f-4b3b-abc5-54f6aa6d43ed" pod="opendatahub/test-curl-11252-5d15eb"
Oct 30 05:12:41.641560 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:41.641515    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11252-aef277" podStartSLOduration=1.641501536 podStartE2EDuration="1.641501536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:41.637981363 +0000 UTC m=+4541.379404457" watchObservedRunningTime="2026-10-30 05:12:41.641501536 +0000 UTC m=+4541.382924633"
Oct 30 05:12:41.652193 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:41.652163    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11252-5d15eb\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="7e383c67-de6f-4b3b-abc5-54f6aa6d43ed" pod="opendatahub/test-curl-11252-5d15eb"
Oct 30 05:12:41.752280 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:41.752246    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:12:42.527855 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:42.527766    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11252-aef277"]
Oct 30 05:12:42.536762 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:42.536728    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11252-aef277"]
Oct 30 05:12:42.603499 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:42.603463    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"]
Oct 30 05:12:42.635604 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:42.635561    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"]
Oct 30 05:12:43.137329 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.137294    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"]
Oct 30 05:12:43.137937 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.137922    5072 cpu_manager.go:414] "containerMap: removing container" podUID="aecb42b8-bb7f-4d92-b894-ecb176176c0c" containerName="test-curl-11252-aef277"
Oct 30 05:12:43.137979 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.137938    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecb42b8-bb7f-4d92-b894-ecb176176c0c" containerName="test-curl-11252-aef277"
Oct 30 05:12:43.138109 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.138098    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecb42b8-bb7f-4d92-b894-ecb176176c0c" containerName="test-curl-11252-aef277"
Oct 30 05:12:43.141786 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.141769    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:43.157316 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.157292    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:12:43.157441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.157412    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:12:43.160623 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.160597    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"]
Oct 30 05:12:43.176304 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.176275    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d97624e5-tt946\" (UID: \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:43.176414 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.176361    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbv5b\" (UniqueName: \"kubernetes.io/projected/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-kube-api-access-fbv5b\") pod \"maas-api-revoke-sub-d97624e5-tt946\" (UID: \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:43.176625 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.176601    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d97624e5-tt946\" (UID: \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:43.279691 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.278423    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbv5b\" (UniqueName: \"kubernetes.io/projected/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-kube-api-access-fbv5b\") pod \"maas-api-revoke-sub-d97624e5-tt946\" (UID: \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:43.280257 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.280229    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d97624e5-tt946\" (UID: \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:43.280489 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.280460    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d97624e5-tt946\" (UID: \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:43.281726 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.281124    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d97624e5-tt946\" (UID: \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:43.286484 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.286458    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d97624e5-tt946\" (UID: \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:43.304064 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.304031    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbv5b\" (UniqueName: \"kubernetes.io/projected/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-kube-api-access-fbv5b\") pod \"maas-api-revoke-sub-d97624e5-tt946\" (UID: \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:43.456046 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.456015    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:43.599982 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.599915    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11252-aef277" podUID="aecb42b8-bb7f-4d92-b894-ecb176176c0c" containerName="test-curl-11252-aef277" containerID="cri-o://5d50fc59113a4c4e5dd7c136b4707345cd0662f516ebd4f39bec5f224935ff73" gracePeriod=2
Oct 30 05:12:43.614305 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:43.614277    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11252-aef277\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="aecb42b8-bb7f-4d92-b894-ecb176176c0c" pod="opendatahub/test-curl-11252-aef277"
Oct 30 05:12:43.709651 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.709583    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11252-5d15eb"
Oct 30 05:12:43.723688 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:43.723636    5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11252-aef277\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="aecb42b8-bb7f-4d92-b894-ecb176176c0c" pod="opendatahub/test-curl-11252-aef277"
Oct 30 05:12:43.756521 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:12:43.756495    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc24ba4_cf9c_465d_b768_e4b5ff10e90d.slice/crio-bd9972269c970fa8a3257997e054e263098e39833591efa79aab765d1340e748 WatchSource:0}: Error finding container bd9972269c970fa8a3257997e054e263098e39833591efa79aab765d1340e748: Status 404 returned error can't find the container with id bd9972269c970fa8a3257997e054e263098e39833591efa79aab765d1340e748
Oct 30 05:12:43.757016 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.756994    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"]
Oct 30 05:12:43.787217 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.787194    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e383c67-de6f-4b3b-abc5-54f6aa6d43ed-kube-api-access-xlb4d\" (UniqueName: \"kubernetes.io/projected/7e383c67-de6f-4b3b-abc5-54f6aa6d43ed-kube-api-access-xlb4d\") pod \"7e383c67-de6f-4b3b-abc5-54f6aa6d43ed\" (UID: \"7e383c67-de6f-4b3b-abc5-54f6aa6d43ed\") "
Oct 30 05:12:43.789272 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.789252    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e383c67-de6f-4b3b-abc5-54f6aa6d43ed-kube-api-access-xlb4d" pod "7e383c67-de6f-4b3b-abc5-54f6aa6d43ed" (UID: "7e383c67-de6f-4b3b-abc5-54f6aa6d43ed"). InnerVolumeSpecName "kube-api-access-xlb4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:43.888972 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:43.888936    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlb4d\" (UniqueName: \"kubernetes.io/projected/7e383c67-de6f-4b3b-abc5-54f6aa6d43ed-kube-api-access-xlb4d\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:44.412288 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:44.412256    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e383c67-de6f-4b3b-abc5-54f6aa6d43ed" path="/var/lib/kubelet/pods/7e383c67-de6f-4b3b-abc5-54f6aa6d43ed/volumes"
Oct 30 05:12:44.610003 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:44.609968    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946" event={"ID":"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d","Type":"ContainerStarted","Data":"861596cb8a0c59bbf2e5049af3dcc12ae9be0d8d2f452d21d65dba6471a7cf95"}
Oct 30 05:12:44.610255 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:44.610018    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946" event={"ID":"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d","Type":"ContainerStarted","Data":"bd9972269c970fa8a3257997e054e263098e39833591efa79aab765d1340e748"}
Oct 30 05:12:44.611404 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:44.611376    5072 generic.go:356] "Generic (PLEG): container finished" podID="7e383c67-de6f-4b3b-abc5-54f6aa6d43ed" containerID="31f553a9d18cc3c876a85d325f0556eb1a04783609296a37e26cd1f4c902e050" exitCode=137
Oct 30 05:12:44.611527 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:44.611425    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11252-5d15eb"
Oct 30 05:12:44.611527 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:44.611449    5072 scope.go:122] "RemoveContainer" containerID="31f553a9d18cc3c876a85d325f0556eb1a04783609296a37e26cd1f4c902e050"
Oct 30 05:12:44.621773 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:44.621747    5072 scope.go:122] "RemoveContainer" containerID="31f553a9d18cc3c876a85d325f0556eb1a04783609296a37e26cd1f4c902e050"
Oct 30 05:12:44.622107 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:44.622058    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f553a9d18cc3c876a85d325f0556eb1a04783609296a37e26cd1f4c902e050\": container with ID starting with 31f553a9d18cc3c876a85d325f0556eb1a04783609296a37e26cd1f4c902e050 not found: ID does not exist" containerID="31f553a9d18cc3c876a85d325f0556eb1a04783609296a37e26cd1f4c902e050"
Oct 30 05:12:44.622189 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:44.622109    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f553a9d18cc3c876a85d325f0556eb1a04783609296a37e26cd1f4c902e050"} err="failed to get container status \"31f553a9d18cc3c876a85d325f0556eb1a04783609296a37e26cd1f4c902e050\": rpc error: code = NotFound desc = could not find container \"31f553a9d18cc3c876a85d325f0556eb1a04783609296a37e26cd1f4c902e050\": container with ID starting with 31f553a9d18cc3c876a85d325f0556eb1a04783609296a37e26cd1f4c902e050 not found: ID does not exist"
Oct 30 05:12:44.652156 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:44.652116    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946" podStartSLOduration=1.652104061 podStartE2EDuration="1.652104061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:44.650614565 +0000 UTC m=+4544.392037787" watchObservedRunningTime="2026-10-30 05:12:44.652104061 +0000 UTC m=+4544.393527158"
Oct 30 05:12:45.621460 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:45.621429    5072 generic.go:356] "Generic (PLEG): container finished" podID="1dc24ba4-cf9c-465d-b768-e4b5ff10e90d" containerID="861596cb8a0c59bbf2e5049af3dcc12ae9be0d8d2f452d21d65dba6471a7cf95" exitCode=0
Oct 30 05:12:45.621764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:45.621507    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946" event={"ID":"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d","Type":"ContainerDied","Data":"861596cb8a0c59bbf2e5049af3dcc12ae9be0d8d2f452d21d65dba6471a7cf95"}
Oct 30 05:12:45.832828 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:45.832795    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11252-aef277"
Oct 30 05:12:45.917825 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:45.917730    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aecb42b8-bb7f-4d92-b894-ecb176176c0c-kube-api-access-6qgrk\" (UniqueName: \"kubernetes.io/projected/aecb42b8-bb7f-4d92-b894-ecb176176c0c-kube-api-access-6qgrk\") pod \"aecb42b8-bb7f-4d92-b894-ecb176176c0c\" (UID: \"aecb42b8-bb7f-4d92-b894-ecb176176c0c\") "
Oct 30 05:12:45.920006 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:45.919978    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecb42b8-bb7f-4d92-b894-ecb176176c0c-kube-api-access-6qgrk" pod "aecb42b8-bb7f-4d92-b894-ecb176176c0c" (UID: "aecb42b8-bb7f-4d92-b894-ecb176176c0c"). InnerVolumeSpecName "kube-api-access-6qgrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:46.018986 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.018946    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qgrk\" (UniqueName: \"kubernetes.io/projected/aecb42b8-bb7f-4d92-b894-ecb176176c0c-kube-api-access-6qgrk\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:46.408266 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.408232    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aecb42b8-bb7f-4d92-b894-ecb176176c0c" path="/var/lib/kubelet/pods/aecb42b8-bb7f-4d92-b894-ecb176176c0c/volumes"
Oct 30 05:12:46.629424 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.629379    5072 generic.go:356] "Generic (PLEG): container finished" podID="aecb42b8-bb7f-4d92-b894-ecb176176c0c" containerID="5d50fc59113a4c4e5dd7c136b4707345cd0662f516ebd4f39bec5f224935ff73" exitCode=137
Oct 30 05:12:46.629893 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.629431    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11252-aef277"
Oct 30 05:12:46.629893 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.629481    5072 scope.go:122] "RemoveContainer" containerID="5d50fc59113a4c4e5dd7c136b4707345cd0662f516ebd4f39bec5f224935ff73"
Oct 30 05:12:46.642710 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.642674    5072 scope.go:122] "RemoveContainer" containerID="5d50fc59113a4c4e5dd7c136b4707345cd0662f516ebd4f39bec5f224935ff73"
Oct 30 05:12:46.643002 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:12:46.642981    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d50fc59113a4c4e5dd7c136b4707345cd0662f516ebd4f39bec5f224935ff73\": container with ID starting with 5d50fc59113a4c4e5dd7c136b4707345cd0662f516ebd4f39bec5f224935ff73 not found: ID does not exist" containerID="5d50fc59113a4c4e5dd7c136b4707345cd0662f516ebd4f39bec5f224935ff73"
Oct 30 05:12:46.643121 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.643012    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d50fc59113a4c4e5dd7c136b4707345cd0662f516ebd4f39bec5f224935ff73"} err="failed to get container status \"5d50fc59113a4c4e5dd7c136b4707345cd0662f516ebd4f39bec5f224935ff73\": rpc error: code = NotFound desc = could not find container \"5d50fc59113a4c4e5dd7c136b4707345cd0662f516ebd4f39bec5f224935ff73\": container with ID starting with 5d50fc59113a4c4e5dd7c136b4707345cd0662f516ebd4f39bec5f224935ff73 not found: ID does not exist"
Oct 30 05:12:46.786609 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.786579    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:46.929232 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.929199    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-kube-api-access-fbv5b\" (UniqueName: \"kubernetes.io/projected/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-kube-api-access-fbv5b\") pod \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\" (UID: \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\") "
Oct 30 05:12:46.929232 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.929236    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-xks-ca\") pod \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\" (UID: \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\") "
Oct 30 05:12:46.929514 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.929385    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-openshift-ca\") pod \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\" (UID: \"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d\") "
Oct 30 05:12:46.929781 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.929752    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-openshift-ca" pod "1dc24ba4-cf9c-465d-b768-e4b5ff10e90d" (UID: "1dc24ba4-cf9c-465d-b768-e4b5ff10e90d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:12:46.930053 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.929882    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:46.931531 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.931502    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-xks-ca" pod "1dc24ba4-cf9c-465d-b768-e4b5ff10e90d" (UID: "1dc24ba4-cf9c-465d-b768-e4b5ff10e90d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:12:46.931658 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:46.931597    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-kube-api-access-fbv5b" pod "1dc24ba4-cf9c-465d-b768-e4b5ff10e90d" (UID: "1dc24ba4-cf9c-465d-b768-e4b5ff10e90d"). InnerVolumeSpecName "kube-api-access-fbv5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:47.030969 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:47.030939    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbv5b\" (UniqueName: \"kubernetes.io/projected/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-kube-api-access-fbv5b\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:47.030969 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:47.030966    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:47.603774 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:47.603736    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t" podUID="64ba6a9f-dabb-4827-bbda-7f89e33de8b8" containerName="payload-processing" containerID="cri-o://28dfc0622c7bbeb42a9a2d5823b1548450912f812a6c339484b6707e3a23868a" gracePeriod=25
Oct 30 05:12:47.635930 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:47.635875    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd" podUID="fe5027a9-d674-4f19-a204-7aeae7393585" containerName="payload-pre-processing" containerID="cri-o://b1e9a62dc86601894e9669577c8853a9a950a6d2cc9bc9ee5be71b88fab48247" gracePeriod=25
Oct 30 05:12:47.637874 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:47.637840    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"
Oct 30 05:12:47.637874 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:47.637854    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946" event={"ID":"1dc24ba4-cf9c-465d-b768-e4b5ff10e90d","Type":"ContainerDied","Data":"bd9972269c970fa8a3257997e054e263098e39833591efa79aab765d1340e748"}
Oct 30 05:12:47.638070 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:47.637914    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd9972269c970fa8a3257997e054e263098e39833591efa79aab765d1340e748"
Oct 30 05:12:48.547557 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:48.547520    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd" podUID="fe5027a9-d674-4f19-a204-7aeae7393585" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:12:49.554330 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:49.554287    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t" podUID="64ba6a9f-dabb-4827-bbda-7f89e33de8b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:12:53.470900 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.470863    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"]
Oct 30 05:12:53.471687 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.471660    5072 cpu_manager.go:414] "containerMap: removing container" podUID="1dc24ba4-cf9c-465d-b768-e4b5ff10e90d" containerName="revoke-keys"
Oct 30 05:12:53.471687 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.471687    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc24ba4-cf9c-465d-b768-e4b5ff10e90d" containerName="revoke-keys"
Oct 30 05:12:53.471913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.471900    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc24ba4-cf9c-465d-b768-e4b5ff10e90d" containerName="revoke-keys"
Oct 30 05:12:53.475878 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.475861    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:53.493960 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.493934    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"]
Oct 30 05:12:53.502536 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.502513    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:12:53.502661 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.502607    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:12:53.608923 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.608889    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91b74399-s2h4n\" (UID: \"db06dc72-0f0f-4630-8372-75ef73295abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:53.609146 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.608959    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91b74399-s2h4n\" (UID: \"db06dc72-0f0f-4630-8372-75ef73295abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:53.609146 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.609056    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8l42\" (UniqueName: \"kubernetes.io/projected/db06dc72-0f0f-4630-8372-75ef73295abd-kube-api-access-b8l42\") pod \"maas-api-revoke-sub-91b74399-s2h4n\" (UID: \"db06dc72-0f0f-4630-8372-75ef73295abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:53.710807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.710774    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91b74399-s2h4n\" (UID: \"db06dc72-0f0f-4630-8372-75ef73295abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:53.710986 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.710827    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91b74399-s2h4n\" (UID: \"db06dc72-0f0f-4630-8372-75ef73295abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:53.710986 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.710869    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8l42\" (UniqueName: \"kubernetes.io/projected/db06dc72-0f0f-4630-8372-75ef73295abd-kube-api-access-b8l42\") pod \"maas-api-revoke-sub-91b74399-s2h4n\" (UID: \"db06dc72-0f0f-4630-8372-75ef73295abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:53.711482 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.711451    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91b74399-s2h4n\" (UID: \"db06dc72-0f0f-4630-8372-75ef73295abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:53.713398 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.713373    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91b74399-s2h4n\" (UID: \"db06dc72-0f0f-4630-8372-75ef73295abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:53.735321 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.735263    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8l42\" (UniqueName: \"kubernetes.io/projected/db06dc72-0f0f-4630-8372-75ef73295abd-kube-api-access-b8l42\") pod \"maas-api-revoke-sub-91b74399-s2h4n\" (UID: \"db06dc72-0f0f-4630-8372-75ef73295abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:53.790174 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:53.790145    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:54.095768 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:54.095739    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"]
Oct 30 05:12:54.096419 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:12:54.096390    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb06dc72_0f0f_4630_8372_75ef73295abd.slice/crio-25058a37d88b2373af14a61146c28d6356a85ccfd8878c75e59b01e017c39c02 WatchSource:0}: Error finding container 25058a37d88b2373af14a61146c28d6356a85ccfd8878c75e59b01e017c39c02: Status 404 returned error can't find the container with id 25058a37d88b2373af14a61146c28d6356a85ccfd8878c75e59b01e017c39c02
Oct 30 05:12:54.700105 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:54.700049    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n" event={"ID":"db06dc72-0f0f-4630-8372-75ef73295abd","Type":"ContainerStarted","Data":"b1c18329c8cff6a5f6ae94ed0d115fcb253fa01ea5e3b006c0677f1b43800503"}
Oct 30 05:12:54.700522 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:54.700133    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n" event={"ID":"db06dc72-0f0f-4630-8372-75ef73295abd","Type":"ContainerStarted","Data":"25058a37d88b2373af14a61146c28d6356a85ccfd8878c75e59b01e017c39c02"}
Oct 30 05:12:54.754531 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:54.754476    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n" podStartSLOduration=1.754460293 podStartE2EDuration="1.754460293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:54.740522261 +0000 UTC m=+4554.481945359" watchObservedRunningTime="2026-10-30 05:12:54.754460293 +0000 UTC m=+4554.495883391"
Oct 30 05:12:55.709244 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:55.709210    5072 generic.go:356] "Generic (PLEG): container finished" podID="db06dc72-0f0f-4630-8372-75ef73295abd" containerID="b1c18329c8cff6a5f6ae94ed0d115fcb253fa01ea5e3b006c0677f1b43800503" exitCode=0
Oct 30 05:12:55.709720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:55.709341    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n" event={"ID":"db06dc72-0f0f-4630-8372-75ef73295abd","Type":"ContainerDied","Data":"b1c18329c8cff6a5f6ae94ed0d115fcb253fa01ea5e3b006c0677f1b43800503"}
Oct 30 05:12:56.874345 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:56.874320    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:57.050392 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:57.050299    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-xks-ca\") pod \"db06dc72-0f0f-4630-8372-75ef73295abd\" (UID: \"db06dc72-0f0f-4630-8372-75ef73295abd\") "
Oct 30 05:12:57.050392 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:57.050356    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-openshift-ca\") pod \"db06dc72-0f0f-4630-8372-75ef73295abd\" (UID: \"db06dc72-0f0f-4630-8372-75ef73295abd\") "
Oct 30 05:12:57.050649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:57.050469    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db06dc72-0f0f-4630-8372-75ef73295abd-kube-api-access-b8l42\" (UniqueName: \"kubernetes.io/projected/db06dc72-0f0f-4630-8372-75ef73295abd-kube-api-access-b8l42\") pod \"db06dc72-0f0f-4630-8372-75ef73295abd\" (UID: \"db06dc72-0f0f-4630-8372-75ef73295abd\") "
Oct 30 05:12:57.050742 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:57.050713    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-openshift-ca" pod "db06dc72-0f0f-4630-8372-75ef73295abd" (UID: "db06dc72-0f0f-4630-8372-75ef73295abd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:12:57.050835 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:57.050821    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:57.052694 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:57.052672    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db06dc72-0f0f-4630-8372-75ef73295abd-kube-api-access-b8l42" pod "db06dc72-0f0f-4630-8372-75ef73295abd" (UID: "db06dc72-0f0f-4630-8372-75ef73295abd"). InnerVolumeSpecName "kube-api-access-b8l42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:57.052824 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:57.052733    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-xks-ca" pod "db06dc72-0f0f-4630-8372-75ef73295abd" (UID: "db06dc72-0f0f-4630-8372-75ef73295abd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:12:57.151742 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:57.151711    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8l42\" (UniqueName: \"kubernetes.io/projected/db06dc72-0f0f-4630-8372-75ef73295abd-kube-api-access-b8l42\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:57.151742 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:57.151741    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db06dc72-0f0f-4630-8372-75ef73295abd-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:12:57.724565 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:57.724533    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"
Oct 30 05:12:57.724723 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:57.724569    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n" event={"ID":"db06dc72-0f0f-4630-8372-75ef73295abd","Type":"ContainerDied","Data":"25058a37d88b2373af14a61146c28d6356a85ccfd8878c75e59b01e017c39c02"}
Oct 30 05:12:57.724723 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:57.724618    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25058a37d88b2373af14a61146c28d6356a85ccfd8878c75e59b01e017c39c02"
Oct 30 05:12:58.547201 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:58.547160    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd" podUID="fe5027a9-d674-4f19-a204-7aeae7393585" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:12:59.554896 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:12:59.554849    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t" podUID="64ba6a9f-dabb-4827-bbda-7f89e33de8b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:13:03.396994 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:03.396961    5072 scope.go:122] "RemoveContainer" containerID="d259ff17b82a7512891f3becb13fe685f8dd6be3c911c5636e518c2b3d13c15c"
Oct 30 05:13:04.372638 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.372605    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"]
Oct 30 05:13:04.373257 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.373243    5072 cpu_manager.go:414] "containerMap: removing container" podUID="db06dc72-0f0f-4630-8372-75ef73295abd" containerName="revoke-keys"
Oct 30 05:13:04.373319 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.373258    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db06dc72-0f0f-4630-8372-75ef73295abd" containerName="revoke-keys"
Oct 30 05:13:04.373382 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.373373    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="db06dc72-0f0f-4630-8372-75ef73295abd" containerName="revoke-keys"
Oct 30 05:13:04.378577 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.378559    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:04.394881 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.394854    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"]
Oct 30 05:13:04.403697 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.403672    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:13:04.404113 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.403814    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:13:04.433834 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.433801    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17974c41-62nw8\" (UID: \"ccf04038-6fbc-4804-a889-5e98468e76f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:04.434050 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.433950    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8sxw\" (UniqueName: \"kubernetes.io/projected/ccf04038-6fbc-4804-a889-5e98468e76f6-kube-api-access-h8sxw\") pod \"maas-api-revoke-sub-17974c41-62nw8\" (UID: \"ccf04038-6fbc-4804-a889-5e98468e76f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:04.434050 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.434003    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17974c41-62nw8\" (UID: \"ccf04038-6fbc-4804-a889-5e98468e76f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:04.536679 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.536637    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17974c41-62nw8\" (UID: \"ccf04038-6fbc-4804-a889-5e98468e76f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:04.536886 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.536708    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8sxw\" (UniqueName: \"kubernetes.io/projected/ccf04038-6fbc-4804-a889-5e98468e76f6-kube-api-access-h8sxw\") pod \"maas-api-revoke-sub-17974c41-62nw8\" (UID: \"ccf04038-6fbc-4804-a889-5e98468e76f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:04.536886 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.536740    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17974c41-62nw8\" (UID: \"ccf04038-6fbc-4804-a889-5e98468e76f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:04.537371 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.537345    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17974c41-62nw8\" (UID: \"ccf04038-6fbc-4804-a889-5e98468e76f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:04.539308 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.539286    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17974c41-62nw8\" (UID: \"ccf04038-6fbc-4804-a889-5e98468e76f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:04.560261 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.560235    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8sxw\" (UniqueName: \"kubernetes.io/projected/ccf04038-6fbc-4804-a889-5e98468e76f6-kube-api-access-h8sxw\") pod \"maas-api-revoke-sub-17974c41-62nw8\" (UID: \"ccf04038-6fbc-4804-a889-5e98468e76f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:04.694530 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.694503    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:04.998950 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:04.998922    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"]
Oct 30 05:13:04.999381 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:13:04.999352    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf04038_6fbc_4804_a889_5e98468e76f6.slice/crio-17431b724da0d97f3bbb2e738d4f9798008bdbb0bc7dde3125e5016dc72ea596 WatchSource:0}: Error finding container 17431b724da0d97f3bbb2e738d4f9798008bdbb0bc7dde3125e5016dc72ea596: Status 404 returned error can't find the container with id 17431b724da0d97f3bbb2e738d4f9798008bdbb0bc7dde3125e5016dc72ea596
Oct 30 05:13:05.788351 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:05.788300    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8" event={"ID":"ccf04038-6fbc-4804-a889-5e98468e76f6","Type":"ContainerStarted","Data":"9bacafa18b102c126f71c499fc4bfc27643ed51dd7d367454d76ec3139971791"}
Oct 30 05:13:05.788889 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:05.788383    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8" event={"ID":"ccf04038-6fbc-4804-a889-5e98468e76f6","Type":"ContainerStarted","Data":"17431b724da0d97f3bbb2e738d4f9798008bdbb0bc7dde3125e5016dc72ea596"}
Oct 30 05:13:05.831113 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:05.831044    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8" podStartSLOduration=1.831030142 podStartE2EDuration="1.831030142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:05.829183026 +0000 UTC m=+4565.570606125" watchObservedRunningTime="2026-10-30 05:13:05.831030142 +0000 UTC m=+4565.572453264"
Oct 30 05:13:06.798526 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:06.798493    5072 generic.go:356] "Generic (PLEG): container finished" podID="ccf04038-6fbc-4804-a889-5e98468e76f6" containerID="9bacafa18b102c126f71c499fc4bfc27643ed51dd7d367454d76ec3139971791" exitCode=0
Oct 30 05:13:06.799027 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:06.798583    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8" event={"ID":"ccf04038-6fbc-4804-a889-5e98468e76f6","Type":"ContainerDied","Data":"9bacafa18b102c126f71c499fc4bfc27643ed51dd7d367454d76ec3139971791"}
Oct 30 05:13:07.811455 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:07.809593    5072 generic.go:356] "Generic (PLEG): container finished" podID="fe5027a9-d674-4f19-a204-7aeae7393585" containerID="b1e9a62dc86601894e9669577c8853a9a950a6d2cc9bc9ee5be71b88fab48247" exitCode=0
Oct 30 05:13:07.811455 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:07.809758    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd" event={"ID":"fe5027a9-d674-4f19-a204-7aeae7393585","Type":"ContainerDied","Data":"b1e9a62dc86601894e9669577c8853a9a950a6d2cc9bc9ee5be71b88fab48247"}
Oct 30 05:13:07.814164 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:07.813755    5072 generic.go:356] "Generic (PLEG): container finished" podID="64ba6a9f-dabb-4827-bbda-7f89e33de8b8" containerID="28dfc0622c7bbeb42a9a2d5823b1548450912f812a6c339484b6707e3a23868a" exitCode=0
Oct 30 05:13:07.814164 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:07.814103    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t" event={"ID":"64ba6a9f-dabb-4827-bbda-7f89e33de8b8","Type":"ContainerDied","Data":"28dfc0622c7bbeb42a9a2d5823b1548450912f812a6c339484b6707e3a23868a"}
Oct 30 05:13:07.929666 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:07.929637    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"
Oct 30 05:13:07.984401 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:07.984373    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"
Oct 30 05:13:08.002750 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.002727    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:08.004883 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.004863    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-plugins-config-volume\") pod \"64ba6a9f-dabb-4827-bbda-7f89e33de8b8\" (UID: \"64ba6a9f-dabb-4827-bbda-7f89e33de8b8\") "
Oct 30 05:13:08.005063 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.005044    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-kube-api-access-55dsp\" (UniqueName: \"kubernetes.io/projected/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-kube-api-access-55dsp\") pod \"64ba6a9f-dabb-4827-bbda-7f89e33de8b8\" (UID: \"64ba6a9f-dabb-4827-bbda-7f89e33de8b8\") "
Oct 30 05:13:08.005315 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.005292    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-plugins-config-volume" pod "64ba6a9f-dabb-4827-bbda-7f89e33de8b8" (UID: "64ba6a9f-dabb-4827-bbda-7f89e33de8b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:08.005585 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.005567    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:08.007447 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.007429    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-kube-api-access-55dsp" pod "64ba6a9f-dabb-4827-bbda-7f89e33de8b8" (UID: "64ba6a9f-dabb-4827-bbda-7f89e33de8b8"). InnerVolumeSpecName "kube-api-access-55dsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:08.106070 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.106039    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5027a9-d674-4f19-a204-7aeae7393585-kube-api-access-dwkml\" (UniqueName: \"kubernetes.io/projected/fe5027a9-d674-4f19-a204-7aeae7393585-kube-api-access-dwkml\") pod \"fe5027a9-d674-4f19-a204-7aeae7393585\" (UID: \"fe5027a9-d674-4f19-a204-7aeae7393585\") "
Oct 30 05:13:08.106271 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.106161    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-openshift-ca\") pod \"ccf04038-6fbc-4804-a889-5e98468e76f6\" (UID: \"ccf04038-6fbc-4804-a889-5e98468e76f6\") "
Oct 30 05:13:08.106271 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.106229    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe5027a9-d674-4f19-a204-7aeae7393585-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5027a9-d674-4f19-a204-7aeae7393585-plugins-config-volume\") pod \"fe5027a9-d674-4f19-a204-7aeae7393585\" (UID: \"fe5027a9-d674-4f19-a204-7aeae7393585\") "
Oct 30 05:13:08.106271 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.106256    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf04038-6fbc-4804-a889-5e98468e76f6-kube-api-access-h8sxw\" (UniqueName: \"kubernetes.io/projected/ccf04038-6fbc-4804-a889-5e98468e76f6-kube-api-access-h8sxw\") pod \"ccf04038-6fbc-4804-a889-5e98468e76f6\" (UID: \"ccf04038-6fbc-4804-a889-5e98468e76f6\") "
Oct 30 05:13:08.106462 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.106284    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-xks-ca\") pod \"ccf04038-6fbc-4804-a889-5e98468e76f6\" (UID: \"ccf04038-6fbc-4804-a889-5e98468e76f6\") "
Oct 30 05:13:08.106823 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.106660    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5027a9-d674-4f19-a204-7aeae7393585-plugins-config-volume" pod "fe5027a9-d674-4f19-a204-7aeae7393585" (UID: "fe5027a9-d674-4f19-a204-7aeae7393585"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:08.106956 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.106832    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-openshift-ca" pod "ccf04038-6fbc-4804-a889-5e98468e76f6" (UID: "ccf04038-6fbc-4804-a889-5e98468e76f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:08.107127 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.107005    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:08.107364 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.107171    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55dsp\" (UniqueName: \"kubernetes.io/projected/64ba6a9f-dabb-4827-bbda-7f89e33de8b8-kube-api-access-55dsp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:08.107364 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.107267    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5027a9-d674-4f19-a204-7aeae7393585-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:08.108688 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.108665    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf04038-6fbc-4804-a889-5e98468e76f6-kube-api-access-h8sxw" pod "ccf04038-6fbc-4804-a889-5e98468e76f6" (UID: "ccf04038-6fbc-4804-a889-5e98468e76f6"). InnerVolumeSpecName "kube-api-access-h8sxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:08.108782 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.108768    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-xks-ca" pod "ccf04038-6fbc-4804-a889-5e98468e76f6" (UID: "ccf04038-6fbc-4804-a889-5e98468e76f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:08.109015 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.108995    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5027a9-d674-4f19-a204-7aeae7393585-kube-api-access-dwkml" pod "fe5027a9-d674-4f19-a204-7aeae7393585" (UID: "fe5027a9-d674-4f19-a204-7aeae7393585"). InnerVolumeSpecName "kube-api-access-dwkml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:08.208619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.208589    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwkml\" (UniqueName: \"kubernetes.io/projected/fe5027a9-d674-4f19-a204-7aeae7393585-kube-api-access-dwkml\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:08.208619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.208617    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8sxw\" (UniqueName: \"kubernetes.io/projected/ccf04038-6fbc-4804-a889-5e98468e76f6-kube-api-access-h8sxw\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:08.208791 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.208627    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf04038-6fbc-4804-a889-5e98468e76f6-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:08.822417 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.822379    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"
Oct 30 05:13:08.822863 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.822412    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd" event={"ID":"fe5027a9-d674-4f19-a204-7aeae7393585","Type":"ContainerDied","Data":"d6f5455804fcdd374ce6a02e5b352f0a78b429de9a64c8df09a5190cfecbf015"}
Oct 30 05:13:08.822863 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.822486    5072 scope.go:122] "RemoveContainer" containerID="b1e9a62dc86601894e9669577c8853a9a950a6d2cc9bc9ee5be71b88fab48247"
Oct 30 05:13:08.823883 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.823863    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"
Oct 30 05:13:08.823968 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.823889    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8" event={"ID":"ccf04038-6fbc-4804-a889-5e98468e76f6","Type":"ContainerDied","Data":"17431b724da0d97f3bbb2e738d4f9798008bdbb0bc7dde3125e5016dc72ea596"}
Oct 30 05:13:08.823968 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.823926    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17431b724da0d97f3bbb2e738d4f9798008bdbb0bc7dde3125e5016dc72ea596"
Oct 30 05:13:08.825617 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.825591    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"
Oct 30 05:13:08.825718 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.825624    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t" event={"ID":"64ba6a9f-dabb-4827-bbda-7f89e33de8b8","Type":"ContainerDied","Data":"661b42ce8793716f1fe800a9da7e2a4f07df9076bd04cbd0b38838b3a21d436e"}
Oct 30 05:13:08.834256 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.834234    5072 scope.go:122] "RemoveContainer" containerID="28dfc0622c7bbeb42a9a2d5823b1548450912f812a6c339484b6707e3a23868a"
Oct 30 05:13:08.875646 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.875590    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"]
Oct 30 05:13:08.884434 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.884401    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4c2415-65fb5f487d-vvt6t"]
Oct 30 05:13:08.919486 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.919464    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"]
Oct 30 05:13:08.924161 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:08.924134    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4c2415-5d69f95ddc-sqdkd"]
Oct 30 05:13:10.408825 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:10.408792    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ba6a9f-dabb-4827-bbda-7f89e33de8b8" path="/var/lib/kubelet/pods/64ba6a9f-dabb-4827-bbda-7f89e33de8b8/volumes"
Oct 30 05:13:10.409364 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:10.409346    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe5027a9-d674-4f19-a204-7aeae7393585" path="/var/lib/kubelet/pods/fe5027a9-d674-4f19-a204-7aeae7393585/volumes"
Oct 30 05:13:11.751651 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:13:11.751614    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:13:14.970966 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:14.970927    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"]
Oct 30 05:13:14.971546 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:14.971532    5072 cpu_manager.go:414] "containerMap: removing container" podUID="64ba6a9f-dabb-4827-bbda-7f89e33de8b8" containerName="payload-processing"
Oct 30 05:13:14.971593 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:14.971547    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ba6a9f-dabb-4827-bbda-7f89e33de8b8" containerName="payload-processing"
Oct 30 05:13:14.971593 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:14.971582    5072 cpu_manager.go:414] "containerMap: removing container" podUID="fe5027a9-d674-4f19-a204-7aeae7393585" containerName="payload-pre-processing"
Oct 30 05:13:14.971593 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:14.971587    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5027a9-d674-4f19-a204-7aeae7393585" containerName="payload-pre-processing"
Oct 30 05:13:14.971700 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:14.971601    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ccf04038-6fbc-4804-a889-5e98468e76f6" containerName="revoke-keys"
Oct 30 05:13:14.971700 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:14.971606    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf04038-6fbc-4804-a889-5e98468e76f6" containerName="revoke-keys"
Oct 30 05:13:14.971700 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:14.971699    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5027a9-d674-4f19-a204-7aeae7393585" containerName="payload-pre-processing"
Oct 30 05:13:14.971793 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:14.971708    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf04038-6fbc-4804-a889-5e98468e76f6" containerName="revoke-keys"
Oct 30 05:13:14.971793 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:14.971718    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ba6a9f-dabb-4827-bbda-7f89e33de8b8" containerName="payload-processing"
Oct 30 05:13:14.976738 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:14.976722    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:14.993557 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:14.993532    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"]
Oct 30 05:13:15.002302 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.002262    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:13:15.002534 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.002496    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:13:15.085938 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.085903    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e27989b-d5crc\" (UID: \"e42bc572-4da4-4491-a459-c6a00e1fb298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:15.086134 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.086047    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwt9x\" (UniqueName: \"kubernetes.io/projected/e42bc572-4da4-4491-a459-c6a00e1fb298-kube-api-access-bwt9x\") pod \"maas-api-revoke-sub-8e27989b-d5crc\" (UID: \"e42bc572-4da4-4491-a459-c6a00e1fb298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:15.086134 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.086111    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e27989b-d5crc\" (UID: \"e42bc572-4da4-4491-a459-c6a00e1fb298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:15.187643 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.187612    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwt9x\" (UniqueName: \"kubernetes.io/projected/e42bc572-4da4-4491-a459-c6a00e1fb298-kube-api-access-bwt9x\") pod \"maas-api-revoke-sub-8e27989b-d5crc\" (UID: \"e42bc572-4da4-4491-a459-c6a00e1fb298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:15.187643 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.187647    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e27989b-d5crc\" (UID: \"e42bc572-4da4-4491-a459-c6a00e1fb298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:15.187864 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.187713    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e27989b-d5crc\" (UID: \"e42bc572-4da4-4491-a459-c6a00e1fb298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:15.188401 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.188374    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e27989b-d5crc\" (UID: \"e42bc572-4da4-4491-a459-c6a00e1fb298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:15.190281 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.190258    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e27989b-d5crc\" (UID: \"e42bc572-4da4-4491-a459-c6a00e1fb298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:15.211789 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.211759    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwt9x\" (UniqueName: \"kubernetes.io/projected/e42bc572-4da4-4491-a459-c6a00e1fb298-kube-api-access-bwt9x\") pod \"maas-api-revoke-sub-8e27989b-d5crc\" (UID: \"e42bc572-4da4-4491-a459-c6a00e1fb298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:15.290411 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.290341    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:15.594686 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.594653    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"]
Oct 30 05:13:15.595288 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:13:15.595259    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42bc572_4da4_4491_a459_c6a00e1fb298.slice/crio-c1a51a137182aaa47b59c74e0b0211bef80c1bca65d1b96ea230f2086f09dc88 WatchSource:0}: Error finding container c1a51a137182aaa47b59c74e0b0211bef80c1bca65d1b96ea230f2086f09dc88: Status 404 returned error can't find the container with id c1a51a137182aaa47b59c74e0b0211bef80c1bca65d1b96ea230f2086f09dc88
Oct 30 05:13:15.878852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.878822    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc" event={"ID":"e42bc572-4da4-4491-a459-c6a00e1fb298","Type":"ContainerStarted","Data":"52576ab0408f0ad81c0434d9d49fa22546178e0d5b515e407aedec68feaaf581"}
Oct 30 05:13:15.878999 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.878882    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc" event={"ID":"e42bc572-4da4-4491-a459-c6a00e1fb298","Type":"ContainerStarted","Data":"c1a51a137182aaa47b59c74e0b0211bef80c1bca65d1b96ea230f2086f09dc88"}
Oct 30 05:13:15.920806 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:15.920756    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc" podStartSLOduration=1.9207429980000001 podStartE2EDuration="1.920742998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:15.919417457 +0000 UTC m=+4575.660840555" watchObservedRunningTime="2026-10-30 05:13:15.920742998 +0000 UTC m=+4575.662166110"
Oct 30 05:13:16.887931 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:16.887843    5072 generic.go:356] "Generic (PLEG): container finished" podID="e42bc572-4da4-4491-a459-c6a00e1fb298" containerID="52576ab0408f0ad81c0434d9d49fa22546178e0d5b515e407aedec68feaaf581" exitCode=0
Oct 30 05:13:16.888347 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:16.887933    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc" event={"ID":"e42bc572-4da4-4491-a459-c6a00e1fb298","Type":"ContainerDied","Data":"52576ab0408f0ad81c0434d9d49fa22546178e0d5b515e407aedec68feaaf581"}
Oct 30 05:13:18.050780 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.050754    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:18.221968 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.221939    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42bc572-4da4-4491-a459-c6a00e1fb298-kube-api-access-bwt9x\" (UniqueName: \"kubernetes.io/projected/e42bc572-4da4-4491-a459-c6a00e1fb298-kube-api-access-bwt9x\") pod \"e42bc572-4da4-4491-a459-c6a00e1fb298\" (UID: \"e42bc572-4da4-4491-a459-c6a00e1fb298\") "
Oct 30 05:13:18.222169 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.221974    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-xks-ca\") pod \"e42bc572-4da4-4491-a459-c6a00e1fb298\" (UID: \"e42bc572-4da4-4491-a459-c6a00e1fb298\") "
Oct 30 05:13:18.222169 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.222045    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-openshift-ca\") pod \"e42bc572-4da4-4491-a459-c6a00e1fb298\" (UID: \"e42bc572-4da4-4491-a459-c6a00e1fb298\") "
Oct 30 05:13:18.222517 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.222485    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-openshift-ca" pod "e42bc572-4da4-4491-a459-c6a00e1fb298" (UID: "e42bc572-4da4-4491-a459-c6a00e1fb298"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:18.222636 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.222621    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:18.224285 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.224254    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42bc572-4da4-4491-a459-c6a00e1fb298-kube-api-access-bwt9x" pod "e42bc572-4da4-4491-a459-c6a00e1fb298" (UID: "e42bc572-4da4-4491-a459-c6a00e1fb298"). InnerVolumeSpecName "kube-api-access-bwt9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:18.224285 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.224264    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-xks-ca" pod "e42bc572-4da4-4491-a459-c6a00e1fb298" (UID: "e42bc572-4da4-4491-a459-c6a00e1fb298"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:18.324206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.324178    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwt9x\" (UniqueName: \"kubernetes.io/projected/e42bc572-4da4-4491-a459-c6a00e1fb298-kube-api-access-bwt9x\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:18.324206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.324204    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42bc572-4da4-4491-a459-c6a00e1fb298-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:18.904361 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.904314    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc" event={"ID":"e42bc572-4da4-4491-a459-c6a00e1fb298","Type":"ContainerDied","Data":"c1a51a137182aaa47b59c74e0b0211bef80c1bca65d1b96ea230f2086f09dc88"}
Oct 30 05:13:18.904546 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.904387    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1a51a137182aaa47b59c74e0b0211bef80c1bca65d1b96ea230f2086f09dc88"
Oct 30 05:13:18.904546 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:18.904401    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"
Oct 30 05:13:31.133147 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.133109    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"]
Oct 30 05:13:31.133751 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.133736    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e42bc572-4da4-4491-a459-c6a00e1fb298" containerName="revoke-keys"
Oct 30 05:13:31.133803 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.133752    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42bc572-4da4-4491-a459-c6a00e1fb298" containerName="revoke-keys"
Oct 30 05:13:31.133911 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.133901    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42bc572-4da4-4491-a459-c6a00e1fb298" containerName="revoke-keys"
Oct 30 05:13:31.137722 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.137701    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:31.155431 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.155402    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"]
Oct 30 05:13:31.163692 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.163669    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:13:31.163838 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.163816    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:13:31.169384 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.169356    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65592f9a-g9jr5\" (UID: \"2cf339b3-44ed-48da-93a0-da34264415e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:31.169506 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.169434    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pctx\" (UniqueName: \"kubernetes.io/projected/2cf339b3-44ed-48da-93a0-da34264415e5-kube-api-access-9pctx\") pod \"maas-api-revoke-sub-65592f9a-g9jr5\" (UID: \"2cf339b3-44ed-48da-93a0-da34264415e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:31.169586 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.169562    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65592f9a-g9jr5\" (UID: \"2cf339b3-44ed-48da-93a0-da34264415e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:31.270691 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.270653    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65592f9a-g9jr5\" (UID: \"2cf339b3-44ed-48da-93a0-da34264415e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:31.270889 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.270733    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pctx\" (UniqueName: \"kubernetes.io/projected/2cf339b3-44ed-48da-93a0-da34264415e5-kube-api-access-9pctx\") pod \"maas-api-revoke-sub-65592f9a-g9jr5\" (UID: \"2cf339b3-44ed-48da-93a0-da34264415e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:31.270889 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.270798    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65592f9a-g9jr5\" (UID: \"2cf339b3-44ed-48da-93a0-da34264415e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:31.271357 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.271334    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65592f9a-g9jr5\" (UID: \"2cf339b3-44ed-48da-93a0-da34264415e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:31.273357 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.273333    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65592f9a-g9jr5\" (UID: \"2cf339b3-44ed-48da-93a0-da34264415e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:31.294310 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.294288    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pctx\" (UniqueName: \"kubernetes.io/projected/2cf339b3-44ed-48da-93a0-da34264415e5-kube-api-access-9pctx\") pod \"maas-api-revoke-sub-65592f9a-g9jr5\" (UID: \"2cf339b3-44ed-48da-93a0-da34264415e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:31.452181 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.452140    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:31.760884 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.760856    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"]
Oct 30 05:13:31.999562 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:31.999525    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5" event={"ID":"2cf339b3-44ed-48da-93a0-da34264415e5","Type":"ContainerStarted","Data":"a1e2aceb3c69ac8f590c42b1e3bc97e3d36e68dac80c1bb6d97d31357fe54ba8"}
Oct 30 05:13:33.007814 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:33.007782    5072 generic.go:356] "Generic (PLEG): container finished" podID="2cf339b3-44ed-48da-93a0-da34264415e5" containerID="b3dbcae6f54cb62d7c77291548e5a6dbe3861447a7fdbdde49af5f61db638d07" exitCode=0
Oct 30 05:13:33.008222 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:33.007857    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5" event={"ID":"2cf339b3-44ed-48da-93a0-da34264415e5","Type":"ContainerDied","Data":"b3dbcae6f54cb62d7c77291548e5a6dbe3861447a7fdbdde49af5f61db638d07"}
Oct 30 05:13:34.188158 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:34.188132    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:34.313348 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:34.313266    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-openshift-ca\") pod \"2cf339b3-44ed-48da-93a0-da34264415e5\" (UID: \"2cf339b3-44ed-48da-93a0-da34264415e5\") "
Oct 30 05:13:34.313348 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:34.313310    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf339b3-44ed-48da-93a0-da34264415e5-kube-api-access-9pctx\" (UniqueName: \"kubernetes.io/projected/2cf339b3-44ed-48da-93a0-da34264415e5-kube-api-access-9pctx\") pod \"2cf339b3-44ed-48da-93a0-da34264415e5\" (UID: \"2cf339b3-44ed-48da-93a0-da34264415e5\") "
Oct 30 05:13:34.313538 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:34.313461    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-xks-ca\") pod \"2cf339b3-44ed-48da-93a0-da34264415e5\" (UID: \"2cf339b3-44ed-48da-93a0-da34264415e5\") "
Oct 30 05:13:34.313614 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:34.313592    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-openshift-ca" pod "2cf339b3-44ed-48da-93a0-da34264415e5" (UID: "2cf339b3-44ed-48da-93a0-da34264415e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:34.313958 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:34.313936    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:34.315619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:34.315598    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-xks-ca" pod "2cf339b3-44ed-48da-93a0-da34264415e5" (UID: "2cf339b3-44ed-48da-93a0-da34264415e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:34.315717 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:34.315699    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf339b3-44ed-48da-93a0-da34264415e5-kube-api-access-9pctx" pod "2cf339b3-44ed-48da-93a0-da34264415e5" (UID: "2cf339b3-44ed-48da-93a0-da34264415e5"). InnerVolumeSpecName "kube-api-access-9pctx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:34.414604 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:34.414567    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf339b3-44ed-48da-93a0-da34264415e5-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:34.414604 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:34.414597    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pctx\" (UniqueName: \"kubernetes.io/projected/2cf339b3-44ed-48da-93a0-da34264415e5-kube-api-access-9pctx\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:35.022130 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:35.022101    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"
Oct 30 05:13:35.022130 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:35.022121    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5" event={"ID":"2cf339b3-44ed-48da-93a0-da34264415e5","Type":"ContainerDied","Data":"a1e2aceb3c69ac8f590c42b1e3bc97e3d36e68dac80c1bb6d97d31357fe54ba8"}
Oct 30 05:13:35.022349 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:35.022167    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1e2aceb3c69ac8f590c42b1e3bc97e3d36e68dac80c1bb6d97d31357fe54ba8"
Oct 30 05:13:35.281704 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:35.281628    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"]
Oct 30 05:13:35.319486 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:35.319456    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"]
Oct 30 05:13:39.722212 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.722177    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"]
Oct 30 05:13:39.723018 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.723002    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2cf339b3-44ed-48da-93a0-da34264415e5" containerName="revoke-keys"
Oct 30 05:13:39.723099 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.723021    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf339b3-44ed-48da-93a0-da34264415e5" containerName="revoke-keys"
Oct 30 05:13:39.723194 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.723181    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf339b3-44ed-48da-93a0-da34264415e5" containerName="revoke-keys"
Oct 30 05:13:39.726983 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.726966    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:39.745478 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.745451    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"]
Oct 30 05:13:39.747841 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.747806    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:13:39.753252 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.753233    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:13:39.779413 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.779386    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23397a6f-a7de-442a-bf57-84599654632c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a59d3779-qnt58\" (UID: \"23397a6f-a7de-442a-bf57-84599654632c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:39.779595 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.779575    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqw6s\" (UniqueName: \"kubernetes.io/projected/23397a6f-a7de-442a-bf57-84599654632c-kube-api-access-tqw6s\") pod \"maas-api-revoke-sub-a59d3779-qnt58\" (UID: \"23397a6f-a7de-442a-bf57-84599654632c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:39.779785 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.779756    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23397a6f-a7de-442a-bf57-84599654632c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a59d3779-qnt58\" (UID: \"23397a6f-a7de-442a-bf57-84599654632c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:39.880718 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.880689    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23397a6f-a7de-442a-bf57-84599654632c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a59d3779-qnt58\" (UID: \"23397a6f-a7de-442a-bf57-84599654632c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:39.880838 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.880735    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23397a6f-a7de-442a-bf57-84599654632c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a59d3779-qnt58\" (UID: \"23397a6f-a7de-442a-bf57-84599654632c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:39.880838 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.880815    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqw6s\" (UniqueName: \"kubernetes.io/projected/23397a6f-a7de-442a-bf57-84599654632c-kube-api-access-tqw6s\") pod \"maas-api-revoke-sub-a59d3779-qnt58\" (UID: \"23397a6f-a7de-442a-bf57-84599654632c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:39.881369 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.881348    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23397a6f-a7de-442a-bf57-84599654632c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a59d3779-qnt58\" (UID: \"23397a6f-a7de-442a-bf57-84599654632c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:39.883377 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.883347    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23397a6f-a7de-442a-bf57-84599654632c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a59d3779-qnt58\" (UID: \"23397a6f-a7de-442a-bf57-84599654632c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:39.905454 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:39.905430    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqw6s\" (UniqueName: \"kubernetes.io/projected/23397a6f-a7de-442a-bf57-84599654632c-kube-api-access-tqw6s\") pod \"maas-api-revoke-sub-a59d3779-qnt58\" (UID: \"23397a6f-a7de-442a-bf57-84599654632c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:40.042691 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:40.042609    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:40.282180 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:40.282125    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc" podUID="93480c6a-fe04-426b-9ee4-cdfd827f2ab9" containerName="payload-processing" containerID="cri-o://5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b" gracePeriod=25
Oct 30 05:13:40.320421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:40.320342    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh" podUID="1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0" containerName="payload-pre-processing" containerID="cri-o://10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254" gracePeriod=25
Oct 30 05:13:40.350019 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:13:40.349986    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23397a6f_a7de_442a_bf57_84599654632c.slice/crio-ecc0acc21597f220e280e8f14e3045d0a9f4bbe7a5da2e55bcffcbf66373de6b WatchSource:0}: Error finding container ecc0acc21597f220e280e8f14e3045d0a9f4bbe7a5da2e55bcffcbf66373de6b: Status 404 returned error can't find the container with id ecc0acc21597f220e280e8f14e3045d0a9f4bbe7a5da2e55bcffcbf66373de6b
Oct 30 05:13:40.350637 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:40.350612    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"]
Oct 30 05:13:41.067433 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.067389    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58" event={"ID":"23397a6f-a7de-442a-bf57-84599654632c","Type":"ContainerStarted","Data":"55316dc830cd1569d31ff2cf7dc23c217e0f3afb923a10f7540288d1562fe091"}
Oct 30 05:13:41.067911 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.067450    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58" event={"ID":"23397a6f-a7de-442a-bf57-84599654632c","Type":"ContainerStarted","Data":"ecc0acc21597f220e280e8f14e3045d0a9f4bbe7a5da2e55bcffcbf66373de6b"}
Oct 30 05:13:41.111358 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.111305    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58" podStartSLOduration=2.11129185 podStartE2EDuration="2.11129185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:41.108323722 +0000 UTC m=+4600.849746820" watchObservedRunningTime="2026-10-30 05:13:41.11129185 +0000 UTC m=+4600.852714987"
Oct 30 05:13:41.481516 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.481475    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"]
Oct 30 05:13:41.485627 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.485607    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:41.548574 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.548542    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"]
Oct 30 05:13:41.606148 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.603371    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37cbbf9c-khfw4\" (UID: \"e030506b-4676-4e1f-831d-f5f9a1209ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:41.606148 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.603423    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37cbbf9c-khfw4\" (UID: \"e030506b-4676-4e1f-831d-f5f9a1209ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:41.606148 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.603461    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2ds\" (UniqueName: \"kubernetes.io/projected/e030506b-4676-4e1f-831d-f5f9a1209ecd-kube-api-access-ml2ds\") pod \"maas-api-revoke-sub-37cbbf9c-khfw4\" (UID: \"e030506b-4676-4e1f-831d-f5f9a1209ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:41.704977 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.704943    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37cbbf9c-khfw4\" (UID: \"e030506b-4676-4e1f-831d-f5f9a1209ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:41.705214 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.704998    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37cbbf9c-khfw4\" (UID: \"e030506b-4676-4e1f-831d-f5f9a1209ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:41.705214 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.705030    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2ds\" (UniqueName: \"kubernetes.io/projected/e030506b-4676-4e1f-831d-f5f9a1209ecd-kube-api-access-ml2ds\") pod \"maas-api-revoke-sub-37cbbf9c-khfw4\" (UID: \"e030506b-4676-4e1f-831d-f5f9a1209ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:41.705988 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.705954    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37cbbf9c-khfw4\" (UID: \"e030506b-4676-4e1f-831d-f5f9a1209ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:41.707590 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.707569    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37cbbf9c-khfw4\" (UID: \"e030506b-4676-4e1f-831d-f5f9a1209ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:41.729814 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.729780    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2ds\" (UniqueName: \"kubernetes.io/projected/e030506b-4676-4e1f-831d-f5f9a1209ecd-kube-api-access-ml2ds\") pod \"maas-api-revoke-sub-37cbbf9c-khfw4\" (UID: \"e030506b-4676-4e1f-831d-f5f9a1209ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:41.754910 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:13:41.754874    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:13:41.799782 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:41.799752    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:42.076144 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:42.076111    5072 generic.go:356] "Generic (PLEG): container finished" podID="23397a6f-a7de-442a-bf57-84599654632c" containerID="55316dc830cd1569d31ff2cf7dc23c217e0f3afb923a10f7540288d1562fe091" exitCode=0
Oct 30 05:13:42.076561 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:42.076204    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58" event={"ID":"23397a6f-a7de-442a-bf57-84599654632c","Type":"ContainerDied","Data":"55316dc830cd1569d31ff2cf7dc23c217e0f3afb923a10f7540288d1562fe091"}
Oct 30 05:13:42.102816 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:13:42.102784    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode030506b_4676_4e1f_831d_f5f9a1209ecd.slice/crio-fede564c7c7de3866b8de5bb01e2e4ad9cc5cb3a3192c382308a66fe2b36aa56 WatchSource:0}: Error finding container fede564c7c7de3866b8de5bb01e2e4ad9cc5cb3a3192c382308a66fe2b36aa56: Status 404 returned error can't find the container with id fede564c7c7de3866b8de5bb01e2e4ad9cc5cb3a3192c382308a66fe2b36aa56
Oct 30 05:13:42.108310 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:42.108285    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"]
Oct 30 05:13:43.085510 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.085476    5072 generic.go:356] "Generic (PLEG): container finished" podID="e030506b-4676-4e1f-831d-f5f9a1209ecd" containerID="152b5146621cb162c60cf3bf74621575c8bf4a55f9c4538ba64a3068d1ceb205" exitCode=0
Oct 30 05:13:43.086000 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.085562    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4" event={"ID":"e030506b-4676-4e1f-831d-f5f9a1209ecd","Type":"ContainerDied","Data":"152b5146621cb162c60cf3bf74621575c8bf4a55f9c4538ba64a3068d1ceb205"}
Oct 30 05:13:43.086000 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.085654    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4" event={"ID":"e030506b-4676-4e1f-831d-f5f9a1209ecd","Type":"ContainerStarted","Data":"fede564c7c7de3866b8de5bb01e2e4ad9cc5cb3a3192c382308a66fe2b36aa56"}
Oct 30 05:13:43.229603 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.229581    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:43.326170 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.326139    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23397a6f-a7de-442a-bf57-84599654632c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23397a6f-a7de-442a-bf57-84599654632c-maas-api-openshift-ca\") pod \"23397a6f-a7de-442a-bf57-84599654632c\" (UID: \"23397a6f-a7de-442a-bf57-84599654632c\") "
Oct 30 05:13:43.326343 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.326177    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23397a6f-a7de-442a-bf57-84599654632c-kube-api-access-tqw6s\" (UniqueName: \"kubernetes.io/projected/23397a6f-a7de-442a-bf57-84599654632c-kube-api-access-tqw6s\") pod \"23397a6f-a7de-442a-bf57-84599654632c\" (UID: \"23397a6f-a7de-442a-bf57-84599654632c\") "
Oct 30 05:13:43.326343 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.326200    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23397a6f-a7de-442a-bf57-84599654632c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23397a6f-a7de-442a-bf57-84599654632c-maas-api-xks-ca\") pod \"23397a6f-a7de-442a-bf57-84599654632c\" (UID: \"23397a6f-a7de-442a-bf57-84599654632c\") "
Oct 30 05:13:43.326599 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.326575    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23397a6f-a7de-442a-bf57-84599654632c-maas-api-openshift-ca" pod "23397a6f-a7de-442a-bf57-84599654632c" (UID: "23397a6f-a7de-442a-bf57-84599654632c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:43.326741 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.326723    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23397a6f-a7de-442a-bf57-84599654632c-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:43.328544 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.328522    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23397a6f-a7de-442a-bf57-84599654632c-maas-api-xks-ca" pod "23397a6f-a7de-442a-bf57-84599654632c" (UID: "23397a6f-a7de-442a-bf57-84599654632c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:43.328650 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.328546    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23397a6f-a7de-442a-bf57-84599654632c-kube-api-access-tqw6s" pod "23397a6f-a7de-442a-bf57-84599654632c" (UID: "23397a6f-a7de-442a-bf57-84599654632c"). InnerVolumeSpecName "kube-api-access-tqw6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:43.428061 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.427984    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqw6s\" (UniqueName: \"kubernetes.io/projected/23397a6f-a7de-442a-bf57-84599654632c-kube-api-access-tqw6s\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:43.428061 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:43.428020    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23397a6f-a7de-442a-bf57-84599654632c-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:44.094126 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.094072    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58" event={"ID":"23397a6f-a7de-442a-bf57-84599654632c","Type":"ContainerDied","Data":"ecc0acc21597f220e280e8f14e3045d0a9f4bbe7a5da2e55bcffcbf66373de6b"}
Oct 30 05:13:44.094531 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.094153    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecc0acc21597f220e280e8f14e3045d0a9f4bbe7a5da2e55bcffcbf66373de6b"
Oct 30 05:13:44.094531 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.094097    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"
Oct 30 05:13:44.135094 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.135045    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"]
Oct 30 05:13:44.264486 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.264453    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:44.337759 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.337727    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-xks-ca\") pod \"e030506b-4676-4e1f-831d-f5f9a1209ecd\" (UID: \"e030506b-4676-4e1f-831d-f5f9a1209ecd\") "
Oct 30 05:13:44.337970 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.337840    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e030506b-4676-4e1f-831d-f5f9a1209ecd-kube-api-access-ml2ds\" (UniqueName: \"kubernetes.io/projected/e030506b-4676-4e1f-831d-f5f9a1209ecd-kube-api-access-ml2ds\") pod \"e030506b-4676-4e1f-831d-f5f9a1209ecd\" (UID: \"e030506b-4676-4e1f-831d-f5f9a1209ecd\") "
Oct 30 05:13:44.337970 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.337960    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-openshift-ca\") pod \"e030506b-4676-4e1f-831d-f5f9a1209ecd\" (UID: \"e030506b-4676-4e1f-831d-f5f9a1209ecd\") "
Oct 30 05:13:44.338402 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.338366    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-openshift-ca" pod "e030506b-4676-4e1f-831d-f5f9a1209ecd" (UID: "e030506b-4676-4e1f-831d-f5f9a1209ecd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:44.342232 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.342204    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-xks-ca" pod "e030506b-4676-4e1f-831d-f5f9a1209ecd" (UID: "e030506b-4676-4e1f-831d-f5f9a1209ecd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:44.342350 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.342233    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e030506b-4676-4e1f-831d-f5f9a1209ecd-kube-api-access-ml2ds" pod "e030506b-4676-4e1f-831d-f5f9a1209ecd" (UID: "e030506b-4676-4e1f-831d-f5f9a1209ecd"). InnerVolumeSpecName "kube-api-access-ml2ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:44.439820 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.439786    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:44.439820 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.439819    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml2ds\" (UniqueName: \"kubernetes.io/projected/e030506b-4676-4e1f-831d-f5f9a1209ecd-kube-api-access-ml2ds\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:44.440028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:44.439835    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e030506b-4676-4e1f-831d-f5f9a1209ecd-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:45.105555 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:45.105524    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"
Oct 30 05:13:45.105555 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:45.105536    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4" event={"ID":"e030506b-4676-4e1f-831d-f5f9a1209ecd","Type":"ContainerDied","Data":"fede564c7c7de3866b8de5bb01e2e4ad9cc5cb3a3192c382308a66fe2b36aa56"}
Oct 30 05:13:45.106167 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:45.105584    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fede564c7c7de3866b8de5bb01e2e4ad9cc5cb3a3192c382308a66fe2b36aa56"
Oct 30 05:13:49.485525 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:49.485473    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh" podUID="1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:13:49.485525 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:49.485508    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc" podUID="93480c6a-fe04-426b-9ee4-cdfd827f2ab9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:13:50.946329 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:50.946288    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"]
Oct 30 05:13:50.947019 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:50.947002    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e030506b-4676-4e1f-831d-f5f9a1209ecd" containerName="revoke-keys"
Oct 30 05:13:50.947067 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:50.947021    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e030506b-4676-4e1f-831d-f5f9a1209ecd" containerName="revoke-keys"
Oct 30 05:13:50.947067 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:50.947048    5072 cpu_manager.go:414] "containerMap: removing container" podUID="23397a6f-a7de-442a-bf57-84599654632c" containerName="revoke-keys"
Oct 30 05:13:50.947067 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:50.947053    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23397a6f-a7de-442a-bf57-84599654632c" containerName="revoke-keys"
Oct 30 05:13:50.947210 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:50.947183    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e030506b-4676-4e1f-831d-f5f9a1209ecd" containerName="revoke-keys"
Oct 30 05:13:50.947210 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:50.947195    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="23397a6f-a7de-442a-bf57-84599654632c" containerName="revoke-keys"
Oct 30 05:13:50.952283 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:50.952265    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:50.967988 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:50.967966    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:13:50.968148 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:50.967994    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:13:50.984878 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:50.984854    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"]
Oct 30 05:13:51.120508 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:51.120472    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a423c4-r5dzc\" (UID: \"7366ae8d-b60d-42bf-942f-16f3c2356fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:51.120720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:51.120627    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a423c4-r5dzc\" (UID: \"7366ae8d-b60d-42bf-942f-16f3c2356fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:51.120720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:51.120682    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4ltp\" (UniqueName: \"kubernetes.io/projected/7366ae8d-b60d-42bf-942f-16f3c2356fef-kube-api-access-l4ltp\") pod \"maas-api-revoke-sub-65a423c4-r5dzc\" (UID: \"7366ae8d-b60d-42bf-942f-16f3c2356fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:51.222147 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:51.222035    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a423c4-r5dzc\" (UID: \"7366ae8d-b60d-42bf-942f-16f3c2356fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:51.222147 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:51.222111    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ltp\" (UniqueName: \"kubernetes.io/projected/7366ae8d-b60d-42bf-942f-16f3c2356fef-kube-api-access-l4ltp\") pod \"maas-api-revoke-sub-65a423c4-r5dzc\" (UID: \"7366ae8d-b60d-42bf-942f-16f3c2356fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:51.222358 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:51.222259    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a423c4-r5dzc\" (UID: \"7366ae8d-b60d-42bf-942f-16f3c2356fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:51.222927 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:51.222906    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a423c4-r5dzc\" (UID: \"7366ae8d-b60d-42bf-942f-16f3c2356fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:51.224823 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:51.224793    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a423c4-r5dzc\" (UID: \"7366ae8d-b60d-42bf-942f-16f3c2356fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:51.245715 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:51.245684    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4ltp\" (UniqueName: \"kubernetes.io/projected/7366ae8d-b60d-42bf-942f-16f3c2356fef-kube-api-access-l4ltp\") pod \"maas-api-revoke-sub-65a423c4-r5dzc\" (UID: \"7366ae8d-b60d-42bf-942f-16f3c2356fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:51.266335 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:51.266302    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:51.586284 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:13:51.586242    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7366ae8d_b60d_42bf_942f_16f3c2356fef.slice/crio-0db49ff78cfe6a03d191f58e6844252585c7d2865ad56c3be3aae4bca4f6fa3e WatchSource:0}: Error finding container 0db49ff78cfe6a03d191f58e6844252585c7d2865ad56c3be3aae4bca4f6fa3e: Status 404 returned error can't find the container with id 0db49ff78cfe6a03d191f58e6844252585c7d2865ad56c3be3aae4bca4f6fa3e
Oct 30 05:13:51.588706 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:51.588679    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"]
Oct 30 05:13:52.157489 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:52.157389    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc" event={"ID":"7366ae8d-b60d-42bf-942f-16f3c2356fef","Type":"ContainerStarted","Data":"98dc15f55254e671a509f9a03773ce740bc8dc2e801b301511f32437d8ce84c8"}
Oct 30 05:13:52.157489 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:52.157475    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc" event={"ID":"7366ae8d-b60d-42bf-942f-16f3c2356fef","Type":"ContainerStarted","Data":"0db49ff78cfe6a03d191f58e6844252585c7d2865ad56c3be3aae4bca4f6fa3e"}
Oct 30 05:13:52.199344 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:52.199291    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc" podStartSLOduration=2.199277917 podStartE2EDuration="2.199277917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:52.197768815 +0000 UTC m=+4611.939191912" watchObservedRunningTime="2026-10-30 05:13:52.199277917 +0000 UTC m=+4611.940701213"
Oct 30 05:13:52.365972 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:52.365935    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"]
Oct 30 05:13:53.166130 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:53.166062    5072 generic.go:356] "Generic (PLEG): container finished" podID="7366ae8d-b60d-42bf-942f-16f3c2356fef" containerID="98dc15f55254e671a509f9a03773ce740bc8dc2e801b301511f32437d8ce84c8" exitCode=0
Oct 30 05:13:53.166587 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:53.166147    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc" event={"ID":"7366ae8d-b60d-42bf-942f-16f3c2356fef","Type":"ContainerDied","Data":"98dc15f55254e671a509f9a03773ce740bc8dc2e801b301511f32437d8ce84c8"}
Oct 30 05:13:54.327899 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:54.327871    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:54.366113 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:54.366066    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7366ae8d-b60d-42bf-942f-16f3c2356fef-kube-api-access-l4ltp\" (UniqueName: \"kubernetes.io/projected/7366ae8d-b60d-42bf-942f-16f3c2356fef-kube-api-access-l4ltp\") pod \"7366ae8d-b60d-42bf-942f-16f3c2356fef\" (UID: \"7366ae8d-b60d-42bf-942f-16f3c2356fef\") "
Oct 30 05:13:54.366265 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:54.366153    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-openshift-ca\") pod \"7366ae8d-b60d-42bf-942f-16f3c2356fef\" (UID: \"7366ae8d-b60d-42bf-942f-16f3c2356fef\") "
Oct 30 05:13:54.366333 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:54.366267    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-xks-ca\") pod \"7366ae8d-b60d-42bf-942f-16f3c2356fef\" (UID: \"7366ae8d-b60d-42bf-942f-16f3c2356fef\") "
Oct 30 05:13:54.366539 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:54.366513    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-openshift-ca" pod "7366ae8d-b60d-42bf-942f-16f3c2356fef" (UID: "7366ae8d-b60d-42bf-942f-16f3c2356fef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:54.366708 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:54.366688    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:54.368366 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:54.368321    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7366ae8d-b60d-42bf-942f-16f3c2356fef-kube-api-access-l4ltp" pod "7366ae8d-b60d-42bf-942f-16f3c2356fef" (UID: "7366ae8d-b60d-42bf-942f-16f3c2356fef"). InnerVolumeSpecName "kube-api-access-l4ltp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:54.368701 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:54.368679    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-xks-ca" pod "7366ae8d-b60d-42bf-942f-16f3c2356fef" (UID: "7366ae8d-b60d-42bf-942f-16f3c2356fef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:54.457148 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:13:54.457120    5072 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7366ae8d_b60d_42bf_942f_16f3c2356fef.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 05:13:54.467479 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:54.467441    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4ltp\" (UniqueName: \"kubernetes.io/projected/7366ae8d-b60d-42bf-942f-16f3c2356fef-kube-api-access-l4ltp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:54.467479 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:54.467478    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366ae8d-b60d-42bf-942f-16f3c2356fef-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:55.186021 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:55.185982    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"
Oct 30 05:13:55.186021 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:55.185997    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc" event={"ID":"7366ae8d-b60d-42bf-942f-16f3c2356fef","Type":"ContainerDied","Data":"0db49ff78cfe6a03d191f58e6844252585c7d2865ad56c3be3aae4bca4f6fa3e"}
Oct 30 05:13:55.186272 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:55.186045    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0db49ff78cfe6a03d191f58e6844252585c7d2865ad56c3be3aae4bca4f6fa3e"
Oct 30 05:13:56.756101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:56.738740    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"]
Oct 30 05:13:56.775916 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:56.775880    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"]
Oct 30 05:13:59.159869 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.159824    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2" podUID="3a053e0d-687d-4926-b1d5-cb40e2c769f9" containerName="main" containerID="cri-o://5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562" gracePeriod=45
Oct 30 05:13:59.437663 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.437640    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 05:13:59.486121 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.486033    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc" podUID="93480c6a-fe04-426b-9ee4-cdfd827f2ab9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:13:59.486280 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.486152    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh" podUID="1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:13:59.533807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.533764    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a053e0d-687d-4926-b1d5-cb40e2c769f9-kube-api-access-kfhrv\" (UniqueName: \"kubernetes.io/projected/3a053e0d-687d-4926-b1d5-cb40e2c769f9-kube-api-access-kfhrv\") pod \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") "
Oct 30 05:13:59.534033 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.533834    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tls-certs\") pod \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") "
Oct 30 05:13:59.534033 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.533872    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tmp-dir\") pod \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") "
Oct 30 05:13:59.534033 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.533895    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-model-cache\") pod \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") "
Oct 30 05:13:59.534033 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.533956    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-dshm\") pod \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") "
Oct 30 05:13:59.534357 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.534038    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-home\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-home\") pod \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\" (UID: \"3a053e0d-687d-4926-b1d5-cb40e2c769f9\") "
Oct 30 05:13:59.534357 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.534288    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tmp-dir" pod "3a053e0d-687d-4926-b1d5-cb40e2c769f9" (UID: "3a053e0d-687d-4926-b1d5-cb40e2c769f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:59.534994 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.534470    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-home" pod "3a053e0d-687d-4926-b1d5-cb40e2c769f9" (UID: "3a053e0d-687d-4926-b1d5-cb40e2c769f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:59.535167 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.535005    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:59.535251 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.535237    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:59.535361 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.535343    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-model-cache" pod "3a053e0d-687d-4926-b1d5-cb40e2c769f9" (UID: "3a053e0d-687d-4926-b1d5-cb40e2c769f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:59.536606 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.536579    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a053e0d-687d-4926-b1d5-cb40e2c769f9-kube-api-access-kfhrv" pod "3a053e0d-687d-4926-b1d5-cb40e2c769f9" (UID: "3a053e0d-687d-4926-b1d5-cb40e2c769f9"). InnerVolumeSpecName "kube-api-access-kfhrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:59.537360 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.537339    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-dshm" pod "3a053e0d-687d-4926-b1d5-cb40e2c769f9" (UID: "3a053e0d-687d-4926-b1d5-cb40e2c769f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:59.537468 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.537358    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tls-certs" pod "3a053e0d-687d-4926-b1d5-cb40e2c769f9" (UID: "3a053e0d-687d-4926-b1d5-cb40e2c769f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:59.636995 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.636963    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfhrv\" (UniqueName: \"kubernetes.io/projected/3a053e0d-687d-4926-b1d5-cb40e2c769f9-kube-api-access-kfhrv\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:59.636995 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.636993    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a053e0d-687d-4926-b1d5-cb40e2c769f9-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:59.636995 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.637004    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:13:59.637252 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:13:59.637012    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a053e0d-687d-4926-b1d5-cb40e2c769f9-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.222620 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.222578    5072 generic.go:356] "Generic (PLEG): container finished" podID="3a053e0d-687d-4926-b1d5-cb40e2c769f9" containerID="5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562" exitCode=0
Oct 30 05:14:00.223068 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.222630    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"
Oct 30 05:14:00.223068 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.222663    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2" event={"ID":"3a053e0d-687d-4926-b1d5-cb40e2c769f9","Type":"ContainerDied","Data":"5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562"}
Oct 30 05:14:00.223068 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.222709    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2" event={"ID":"3a053e0d-687d-4926-b1d5-cb40e2c769f9","Type":"ContainerDied","Data":"8f07f9360ff2530673efc4ef99b7a55aaba7a9effc6673e21babf586d821f483"}
Oct 30 05:14:00.223068 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.222728    5072 scope.go:122] "RemoveContainer" containerID="5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562"
Oct 30 05:14:00.234320 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.234301    5072 scope.go:122] "RemoveContainer" containerID="5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562"
Oct 30 05:14:00.234623 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:14:00.234602    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562\": container with ID starting with 5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562 not found: ID does not exist" containerID="5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562"
Oct 30 05:14:00.234692 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.234633    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562"} err="failed to get container status \"5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562\": rpc error: code = NotFound desc = could not find container \"5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562\": container with ID starting with 5a941d596e0d99749df9e59409c8d351989ac0c34cc89dfc164e5c8199bc2562 not found: ID does not exist"
Oct 30 05:14:00.277046 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.277019    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"]
Oct 30 05:14:00.289694 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.289671    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/premium-w4-a2f984-kserve-68495d4cc6-ltlb2"]
Oct 30 05:14:00.408767 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.408737    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a053e0d-687d-4926-b1d5-cb40e2c769f9" path="/var/lib/kubelet/pods/3a053e0d-687d-4926-b1d5-cb40e2c769f9/volumes"
Oct 30 05:14:00.591053 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.591022    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"
Oct 30 05:14:00.621240 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.621215    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"
Oct 30 05:14:00.647156 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.647128    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-kube-api-access-zlwrh\" (UniqueName: \"kubernetes.io/projected/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-kube-api-access-zlwrh\") pod \"93480c6a-fe04-426b-9ee4-cdfd827f2ab9\" (UID: \"93480c6a-fe04-426b-9ee4-cdfd827f2ab9\") "
Oct 30 05:14:00.647340 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.647249    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-plugins-config-volume\") pod \"93480c6a-fe04-426b-9ee4-cdfd827f2ab9\" (UID: \"93480c6a-fe04-426b-9ee4-cdfd827f2ab9\") "
Oct 30 05:14:00.647699 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.647670    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-plugins-config-volume" pod "93480c6a-fe04-426b-9ee4-cdfd827f2ab9" (UID: "93480c6a-fe04-426b-9ee4-cdfd827f2ab9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:00.647899 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.647854    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.649521 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.649497    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-kube-api-access-zlwrh" pod "93480c6a-fe04-426b-9ee4-cdfd827f2ab9" (UID: "93480c6a-fe04-426b-9ee4-cdfd827f2ab9"). InnerVolumeSpecName "kube-api-access-zlwrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:00.748513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.748435    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-kube-api-access-wx4zd\" (UniqueName: \"kubernetes.io/projected/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-kube-api-access-wx4zd\") pod \"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0\" (UID: \"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0\") "
Oct 30 05:14:00.748513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.748506    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-plugins-config-volume\") pod \"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0\" (UID: \"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0\") "
Oct 30 05:14:00.748806 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.748794    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlwrh\" (UniqueName: \"kubernetes.io/projected/93480c6a-fe04-426b-9ee4-cdfd827f2ab9-kube-api-access-zlwrh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.748862 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.748842    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-plugins-config-volume" pod "1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0" (UID: "1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:00.750699 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.750680    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-kube-api-access-wx4zd" pod "1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0" (UID: "1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0"). InnerVolumeSpecName "kube-api-access-wx4zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:00.850442 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.850411    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx4zd\" (UniqueName: \"kubernetes.io/projected/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-kube-api-access-wx4zd\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.850442 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:00.850436    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:01.232471 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.232439    5072 generic.go:356] "Generic (PLEG): container finished" podID="93480c6a-fe04-426b-9ee4-cdfd827f2ab9" containerID="5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b" exitCode=0
Oct 30 05:14:01.232967 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.232482    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"
Oct 30 05:14:01.232967 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.232521    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc" event={"ID":"93480c6a-fe04-426b-9ee4-cdfd827f2ab9","Type":"ContainerDied","Data":"5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b"}
Oct 30 05:14:01.232967 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.232572    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc" event={"ID":"93480c6a-fe04-426b-9ee4-cdfd827f2ab9","Type":"ContainerDied","Data":"fc86e527c6f1b0c32f98ae8969e44a19ec72705b09ef843b494c70dfa924144f"}
Oct 30 05:14:01.232967 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.232600    5072 scope.go:122] "RemoveContainer" containerID="5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b"
Oct 30 05:14:01.234173 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.234029    5072 generic.go:356] "Generic (PLEG): container finished" podID="1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0" containerID="10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254" exitCode=0
Oct 30 05:14:01.234173 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.234071    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"
Oct 30 05:14:01.234173 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.234118    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh" event={"ID":"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0","Type":"ContainerDied","Data":"10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254"}
Oct 30 05:14:01.234173 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.234171    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh" event={"ID":"1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0","Type":"ContainerDied","Data":"ea32d689bda567064bbd7474ead066d161b6976ae095df048602852188db07e5"}
Oct 30 05:14:01.246421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.246401    5072 scope.go:122] "RemoveContainer" containerID="5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b"
Oct 30 05:14:01.246722 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:14:01.246703    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b\": container with ID starting with 5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b not found: ID does not exist" containerID="5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b"
Oct 30 05:14:01.246793 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.246731    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b"} err="failed to get container status \"5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b\": rpc error: code = NotFound desc = could not find container \"5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b\": container with ID starting with 5821b84ccfc489b8567727a87173241f20ca717cf0027a07c17e6f250ed49e8b not found: ID does not exist"
Oct 30 05:14:01.246793 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.246751    5072 scope.go:122] "RemoveContainer" containerID="10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254"
Oct 30 05:14:01.256702 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.256681    5072 scope.go:122] "RemoveContainer" containerID="10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254"
Oct 30 05:14:01.256983 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:14:01.256965    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254\": container with ID starting with 10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254 not found: ID does not exist" containerID="10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254"
Oct 30 05:14:01.257116 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.256990    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254"} err="failed to get container status \"10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254\": rpc error: code = NotFound desc = could not find container \"10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254\": container with ID starting with 10925ea8b3894f4d987dfa4c77f270033e5da941d6a82ceca91ddf4083964254 not found: ID does not exist"
Oct 30 05:14:01.295199 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.295160    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"]
Oct 30 05:14:01.298593 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.298566    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0e964-7458d5f998-692gc"]
Oct 30 05:14:01.333592 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.333555    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"]
Oct 30 05:14:01.337699 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.337672    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0e964-d7d6c8c8c-xgsmh"]
Oct 30 05:14:01.739679 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.739636    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb" podUID="b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a" containerName="payload-processing" containerID="cri-o://07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a" gracePeriod=25
Oct 30 05:14:01.776847 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:01.776799    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n" podUID="bf5d06ae-de18-4fef-8538-477be6f43278" containerName="payload-pre-processing" containerID="cri-o://5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b" gracePeriod=25
Oct 30 05:14:02.354742 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:02.354709    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"]
Oct 30 05:14:02.388122 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:02.388066    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"]
Oct 30 05:14:02.408471 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:02.408442    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0" path="/var/lib/kubelet/pods/1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0/volumes"
Oct 30 05:14:02.408984 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:02.408966    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93480c6a-fe04-426b-9ee4-cdfd827f2ab9" path="/var/lib/kubelet/pods/93480c6a-fe04-426b-9ee4-cdfd827f2ab9/volumes"
Oct 30 05:14:03.450687 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:03.450660    5072 scope.go:122] "RemoveContainer" containerID="aef5e6adce6565067f248fa3d4d96ab9ecd28f0bff30ae71265a39f0efc930df"
Oct 30 05:14:06.140897 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:06.140854    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n" podUID="bf5d06ae-de18-4fef-8538-477be6f43278" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:06.141332 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:06.140901    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb" podUID="b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:07.355427 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.355385    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb" podUID="75405bbd-d694-4d30-9c5b-122843efcd55" containerName="payload-processing" containerID="cri-o://5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160" gracePeriod=25
Oct 30 05:14:07.388388 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.388344    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6" podUID="23dd655b-f213-4e21-8586-7413a7bea249" containerName="payload-pre-processing" containerID="cri-o://388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44" gracePeriod=25
Oct 30 05:14:07.388693 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.388656    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl" podUID="e7105916-eb76-4348-86d3-820f79627195" containerName="main" containerID="cri-o://de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178" gracePeriod=45
Oct 30 05:14:07.669429 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.669399    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:14:07.827552 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.827520    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-model-cache\") pod \"e7105916-eb76-4348-86d3-820f79627195\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") "
Oct 30 05:14:07.827706 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.827584    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7105916-eb76-4348-86d3-820f79627195-kube-api-access-jhhkz\" (UniqueName: \"kubernetes.io/projected/e7105916-eb76-4348-86d3-820f79627195-kube-api-access-jhhkz\") pod \"e7105916-eb76-4348-86d3-820f79627195\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") "
Oct 30 05:14:07.827706 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.827618    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-tmp-dir\") pod \"e7105916-eb76-4348-86d3-820f79627195\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") "
Oct 30 05:14:07.827706 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.827696    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-dshm\") pod \"e7105916-eb76-4348-86d3-820f79627195\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") "
Oct 30 05:14:07.827913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.827725    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-home\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-home\") pod \"e7105916-eb76-4348-86d3-820f79627195\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") "
Oct 30 05:14:07.827913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.827779    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-model-cache" pod "e7105916-eb76-4348-86d3-820f79627195" (UID: "e7105916-eb76-4348-86d3-820f79627195"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:07.827913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.827818    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7105916-eb76-4348-86d3-820f79627195-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7105916-eb76-4348-86d3-820f79627195-tls-certs\") pod \"e7105916-eb76-4348-86d3-820f79627195\" (UID: \"e7105916-eb76-4348-86d3-820f79627195\") "
Oct 30 05:14:07.828051 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.827963    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-tmp-dir" pod "e7105916-eb76-4348-86d3-820f79627195" (UID: "e7105916-eb76-4348-86d3-820f79627195"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:07.828051 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.827993    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-home" pod "e7105916-eb76-4348-86d3-820f79627195" (UID: "e7105916-eb76-4348-86d3-820f79627195"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:07.828432 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.828359    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:07.828432 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.828376    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:07.828432 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.828385    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:07.829923 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.829895    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-dshm" pod "e7105916-eb76-4348-86d3-820f79627195" (UID: "e7105916-eb76-4348-86d3-820f79627195"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:07.830011 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.829969    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7105916-eb76-4348-86d3-820f79627195-tls-certs" pod "e7105916-eb76-4348-86d3-820f79627195" (UID: "e7105916-eb76-4348-86d3-820f79627195"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:14:07.830011 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.829997    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7105916-eb76-4348-86d3-820f79627195-kube-api-access-jhhkz" pod "e7105916-eb76-4348-86d3-820f79627195" (UID: "e7105916-eb76-4348-86d3-820f79627195"). InnerVolumeSpecName "kube-api-access-jhhkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:07.929791 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.929717    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7105916-eb76-4348-86d3-820f79627195-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:07.929791 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.929745    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7105916-eb76-4348-86d3-820f79627195-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:07.929791 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:07.929754    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhhkz\" (UniqueName: \"kubernetes.io/projected/e7105916-eb76-4348-86d3-820f79627195-kube-api-access-jhhkz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:08.286304 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:08.286269    5072 generic.go:356] "Generic (PLEG): container finished" podID="e7105916-eb76-4348-86d3-820f79627195" containerID="de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178" exitCode=0
Oct 30 05:14:08.286454 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:08.286334    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"
Oct 30 05:14:08.286454 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:08.286352    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl" event={"ID":"e7105916-eb76-4348-86d3-820f79627195","Type":"ContainerDied","Data":"de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178"}
Oct 30 05:14:08.286454 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:08.286398    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl" event={"ID":"e7105916-eb76-4348-86d3-820f79627195","Type":"ContainerDied","Data":"6c7572490d610c90113f62c5db9e411e867fcecc1e15e9bfe6b00fc3d843311f"}
Oct 30 05:14:08.286454 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:08.286444    5072 scope.go:122] "RemoveContainer" containerID="de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178"
Oct 30 05:14:08.297736 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:08.297715    5072 scope.go:122] "RemoveContainer" containerID="de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178"
Oct 30 05:14:08.297996 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:14:08.297974    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178\": container with ID starting with de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178 not found: ID does not exist" containerID="de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178"
Oct 30 05:14:08.298062 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:08.298007    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178"} err="failed to get container status \"de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178\": rpc error: code = NotFound desc = could not find container \"de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178\": container with ID starting with de7b4998eed4be91f44b2735d39d870471174d4c4fd94485acc950214c4ae178 not found: ID does not exist"
Oct 30 05:14:08.340875 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:08.340842    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"]
Oct 30 05:14:08.353220 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:08.353200    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e070ea/rate-test-model-e070ea-kserve-6cdcfffb9c-gbwgl"]
Oct 30 05:14:08.408178 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:08.408154    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7105916-eb76-4348-86d3-820f79627195" path="/var/lib/kubelet/pods/e7105916-eb76-4348-86d3-820f79627195/volumes"
Oct 30 05:14:11.751970 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:14:11.751889    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:14:15.434626 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:15.434574    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6" podUID="23dd655b-f213-4e21-8586-7413a7bea249" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:15.435145 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:15.434631    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb" podUID="75405bbd-d694-4d30-9c5b-122843efcd55" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:16.140999 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:16.140952    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n" podUID="bf5d06ae-de18-4fef-8538-477be6f43278" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:16.140999 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:16.140973    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb" podUID="b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:18.677993 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:18.677954    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"]
Oct 30 05:14:22.059274 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.059248    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"
Oct 30 05:14:22.081225 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.081203    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"
Oct 30 05:14:22.099204 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.099182    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-kube-api-access-96g52\" (UniqueName: \"kubernetes.io/projected/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-kube-api-access-96g52\") pod \"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a\" (UID: \"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a\") "
Oct 30 05:14:22.099325 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.099256    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-plugins-config-volume\") pod \"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a\" (UID: \"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a\") "
Oct 30 05:14:22.099819 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.099789    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-plugins-config-volume" pod "b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a" (UID: "b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:22.102369 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.102343    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-kube-api-access-96g52" pod "b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a" (UID: "b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a"). InnerVolumeSpecName "kube-api-access-96g52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:22.200963 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.200928    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf5d06ae-de18-4fef-8538-477be6f43278-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d06ae-de18-4fef-8538-477be6f43278-plugins-config-volume\") pod \"bf5d06ae-de18-4fef-8538-477be6f43278\" (UID: \"bf5d06ae-de18-4fef-8538-477be6f43278\") "
Oct 30 05:14:22.201198 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.200976    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5d06ae-de18-4fef-8538-477be6f43278-kube-api-access-4xc4z\" (UniqueName: \"kubernetes.io/projected/bf5d06ae-de18-4fef-8538-477be6f43278-kube-api-access-4xc4z\") pod \"bf5d06ae-de18-4fef-8538-477be6f43278\" (UID: \"bf5d06ae-de18-4fef-8538-477be6f43278\") "
Oct 30 05:14:22.201414 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.201389    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5d06ae-de18-4fef-8538-477be6f43278-plugins-config-volume" pod "bf5d06ae-de18-4fef-8538-477be6f43278" (UID: "bf5d06ae-de18-4fef-8538-477be6f43278"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:22.201538 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.201521    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d06ae-de18-4fef-8538-477be6f43278-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:22.201593 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.201544    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96g52\" (UniqueName: \"kubernetes.io/projected/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-kube-api-access-96g52\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:22.201593 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.201558    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:22.203316 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.203290    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5d06ae-de18-4fef-8538-477be6f43278-kube-api-access-4xc4z" pod "bf5d06ae-de18-4fef-8538-477be6f43278" (UID: "bf5d06ae-de18-4fef-8538-477be6f43278"). InnerVolumeSpecName "kube-api-access-4xc4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:22.302301 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.302229    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xc4z\" (UniqueName: \"kubernetes.io/projected/bf5d06ae-de18-4fef-8538-477be6f43278-kube-api-access-4xc4z\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:22.383992 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.383959    5072 generic.go:356] "Generic (PLEG): container finished" podID="bf5d06ae-de18-4fef-8538-477be6f43278" containerID="5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b" exitCode=0
Oct 30 05:14:22.384167 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.384009    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"
Oct 30 05:14:22.384167 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.384056    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n" event={"ID":"bf5d06ae-de18-4fef-8538-477be6f43278","Type":"ContainerDied","Data":"5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b"}
Oct 30 05:14:22.384377 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.384182    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n" event={"ID":"bf5d06ae-de18-4fef-8538-477be6f43278","Type":"ContainerDied","Data":"c71d3f786bf8dc7827f352304c38ed200af9f73e81e7339918077439b2458da8"}
Oct 30 05:14:22.384377 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.384220    5072 scope.go:122] "RemoveContainer" containerID="5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b"
Oct 30 05:14:22.385530 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.385508    5072 generic.go:356] "Generic (PLEG): container finished" podID="b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a" containerID="07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a" exitCode=0
Oct 30 05:14:22.385632 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.385536    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb" event={"ID":"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a","Type":"ContainerDied","Data":"07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a"}
Oct 30 05:14:22.385632 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.385569    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"
Oct 30 05:14:22.385632 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.385587    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb" event={"ID":"b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a","Type":"ContainerDied","Data":"ee79eeb0816af84cb525ee72fcf9815c1ab8e4c4c248b26aa98f6523182fe27f"}
Oct 30 05:14:22.397913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.397892    5072 scope.go:122] "RemoveContainer" containerID="5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b"
Oct 30 05:14:22.398197 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:14:22.398176    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b\": container with ID starting with 5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b not found: ID does not exist" containerID="5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b"
Oct 30 05:14:22.398297 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.398204    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b"} err="failed to get container status \"5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b\": rpc error: code = NotFound desc = could not find container \"5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b\": container with ID starting with 5e22e06cb2b95bf91456f05047a2b2cb3523c96b02413f546301cbaec55ff79b not found: ID does not exist"
Oct 30 05:14:22.398297 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.398228    5072 scope.go:122] "RemoveContainer" containerID="07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a"
Oct 30 05:14:22.408572 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.408558    5072 scope.go:122] "RemoveContainer" containerID="07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a"
Oct 30 05:14:22.408817 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:14:22.408799    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a\": container with ID starting with 07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a not found: ID does not exist" containerID="07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a"
Oct 30 05:14:22.408875 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.408823    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a"} err="failed to get container status \"07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a\": rpc error: code = NotFound desc = could not find container \"07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a\": container with ID starting with 07de2ce1b875cc8d26fc1fbc099ea79bd2fcbff6388843e8f3990bad3920f09a not found: ID does not exist"
Oct 30 05:14:22.440115 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.440093    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"]
Oct 30 05:14:22.451905 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.451886    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-91705f-bd7b85d99-97w7n"]
Oct 30 05:14:22.486630 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.486605    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"]
Oct 30 05:14:22.490154 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:22.490127    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-91705f-f6f6f96c8-jq4lb"]
Oct 30 05:14:23.678555 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:23.678511    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4" podUID="f8ff9e15-586c-4b92-9e47-7d79149ebe93" containerName="payload-processing" containerID="cri-o://f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab" gracePeriod=25
Oct 30 05:14:24.432686 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:24.432648    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a" path="/var/lib/kubelet/pods/b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a/volumes"
Oct 30 05:14:24.433677 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:24.433258    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5d06ae-de18-4fef-8538-477be6f43278" path="/var/lib/kubelet/pods/bf5d06ae-de18-4fef-8538-477be6f43278/volumes"
Oct 30 05:14:25.434371 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:25.434331    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6" podUID="23dd655b-f213-4e21-8586-7413a7bea249" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:25.434371 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:25.434350    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb" podUID="75405bbd-d694-4d30-9c5b-122843efcd55" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:27.068311 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.068267    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4" podUID="f8ff9e15-586c-4b92-9e47-7d79149ebe93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:27.669304 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.669277    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"
Oct 30 05:14:27.706929 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.706905    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"
Oct 30 05:14:27.770610 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.770577    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23dd655b-f213-4e21-8586-7413a7bea249-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23dd655b-f213-4e21-8586-7413a7bea249-plugins-config-volume\") pod \"23dd655b-f213-4e21-8586-7413a7bea249\" (UID: \"23dd655b-f213-4e21-8586-7413a7bea249\") "
Oct 30 05:14:27.770791 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.770642    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23dd655b-f213-4e21-8586-7413a7bea249-kube-api-access-prvsp\" (UniqueName: \"kubernetes.io/projected/23dd655b-f213-4e21-8586-7413a7bea249-kube-api-access-prvsp\") pod \"23dd655b-f213-4e21-8586-7413a7bea249\" (UID: \"23dd655b-f213-4e21-8586-7413a7bea249\") "
Oct 30 05:14:27.770791 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.770678    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75405bbd-d694-4d30-9c5b-122843efcd55-kube-api-access-h7hfz\" (UniqueName: \"kubernetes.io/projected/75405bbd-d694-4d30-9c5b-122843efcd55-kube-api-access-h7hfz\") pod \"75405bbd-d694-4d30-9c5b-122843efcd55\" (UID: \"75405bbd-d694-4d30-9c5b-122843efcd55\") "
Oct 30 05:14:27.770791 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.770739    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75405bbd-d694-4d30-9c5b-122843efcd55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75405bbd-d694-4d30-9c5b-122843efcd55-plugins-config-volume\") pod \"75405bbd-d694-4d30-9c5b-122843efcd55\" (UID: \"75405bbd-d694-4d30-9c5b-122843efcd55\") "
Oct 30 05:14:27.771208 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.771179    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23dd655b-f213-4e21-8586-7413a7bea249-plugins-config-volume" pod "23dd655b-f213-4e21-8586-7413a7bea249" (UID: "23dd655b-f213-4e21-8586-7413a7bea249"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:27.771405 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.771217    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75405bbd-d694-4d30-9c5b-122843efcd55-plugins-config-volume" pod "75405bbd-d694-4d30-9c5b-122843efcd55" (UID: "75405bbd-d694-4d30-9c5b-122843efcd55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:27.773310 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.773283    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23dd655b-f213-4e21-8586-7413a7bea249-kube-api-access-prvsp" pod "23dd655b-f213-4e21-8586-7413a7bea249" (UID: "23dd655b-f213-4e21-8586-7413a7bea249"). InnerVolumeSpecName "kube-api-access-prvsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:27.773436 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.773312    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75405bbd-d694-4d30-9c5b-122843efcd55-kube-api-access-h7hfz" pod "75405bbd-d694-4d30-9c5b-122843efcd55" (UID: "75405bbd-d694-4d30-9c5b-122843efcd55"). InnerVolumeSpecName "kube-api-access-h7hfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:27.872577 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.872495    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75405bbd-d694-4d30-9c5b-122843efcd55-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:27.872577 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.872525    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23dd655b-f213-4e21-8586-7413a7bea249-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:27.872577 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.872539    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prvsp\" (UniqueName: \"kubernetes.io/projected/23dd655b-f213-4e21-8586-7413a7bea249-kube-api-access-prvsp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:27.872577 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:27.872553    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7hfz\" (UniqueName: \"kubernetes.io/projected/75405bbd-d694-4d30-9c5b-122843efcd55-kube-api-access-h7hfz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:28.463523 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.463491    5072 generic.go:356] "Generic (PLEG): container finished" podID="75405bbd-d694-4d30-9c5b-122843efcd55" containerID="5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160" exitCode=0
Oct 30 05:14:28.463914 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.463548    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"
Oct 30 05:14:28.463914 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.463571    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb" event={"ID":"75405bbd-d694-4d30-9c5b-122843efcd55","Type":"ContainerDied","Data":"5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160"}
Oct 30 05:14:28.463914 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.463655    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb" event={"ID":"75405bbd-d694-4d30-9c5b-122843efcd55","Type":"ContainerDied","Data":"4d90d13e55daa7958cfc3ddd09055e62989beefe6cc00bdf834dfc7af00f50e0"}
Oct 30 05:14:28.463914 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.463683    5072 scope.go:122] "RemoveContainer" containerID="5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160"
Oct 30 05:14:28.465280 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.465257    5072 generic.go:356] "Generic (PLEG): container finished" podID="23dd655b-f213-4e21-8586-7413a7bea249" containerID="388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44" exitCode=0
Oct 30 05:14:28.465688 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.465287    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6" event={"ID":"23dd655b-f213-4e21-8586-7413a7bea249","Type":"ContainerDied","Data":"388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44"}
Oct 30 05:14:28.465810 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.465709    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6" event={"ID":"23dd655b-f213-4e21-8586-7413a7bea249","Type":"ContainerDied","Data":"e1f201cb4285f952ba10549aec7b4e4ca52555a428c3f2ed91d4a5c601034c21"}
Oct 30 05:14:28.465810 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.465310    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"
Oct 30 05:14:28.474449 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.474434    5072 scope.go:122] "RemoveContainer" containerID="5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160"
Oct 30 05:14:28.474720 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:14:28.474699    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160\": container with ID starting with 5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160 not found: ID does not exist" containerID="5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160"
Oct 30 05:14:28.474783 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.474726    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160"} err="failed to get container status \"5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160\": rpc error: code = NotFound desc = could not find container \"5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160\": container with ID starting with 5f14b85d643bfd17e987bfeb45698bd74e622a8ac3dfeffa6c006509ec668160 not found: ID does not exist"
Oct 30 05:14:28.474783 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.474744    5072 scope.go:122] "RemoveContainer" containerID="388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44"
Oct 30 05:14:28.484619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.484601    5072 scope.go:122] "RemoveContainer" containerID="388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44"
Oct 30 05:14:28.484870 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:14:28.484853    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44\": container with ID starting with 388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44 not found: ID does not exist" containerID="388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44"
Oct 30 05:14:28.484922 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.484876    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44"} err="failed to get container status \"388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44\": rpc error: code = NotFound desc = could not find container \"388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44\": container with ID starting with 388aaf6f43668fd893f5b7addf91777d1733308e541adde2cbe90d63b1790b44 not found: ID does not exist"
Oct 30 05:14:28.518176 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.518153    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"]
Oct 30 05:14:28.531161 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.531127    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-05bee8-7b8ffc66f8-tlzvb"]
Oct 30 05:14:28.566119 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.566097    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"]
Oct 30 05:14:28.572282 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:28.572263    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-05bee8-5fd587974c-r47k6"]
Oct 30 05:14:30.407240 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:30.407205    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23dd655b-f213-4e21-8586-7413a7bea249" path="/var/lib/kubelet/pods/23dd655b-f213-4e21-8586-7413a7bea249/volumes"
Oct 30 05:14:30.407663 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:30.407583    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75405bbd-d694-4d30-9c5b-122843efcd55" path="/var/lib/kubelet/pods/75405bbd-d694-4d30-9c5b-122843efcd55/volumes"
Oct 30 05:14:37.067891 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:37.067855    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4" podUID="f8ff9e15-586c-4b92-9e47-7d79149ebe93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:39.498007 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:39.497950    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"]
Oct 30 05:14:39.498601 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:39.498250    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h" podUID="bf9d8933-4567-4682-bbfa-5a459474528b" containerName="istio-proxy" containerID="cri-o://2a0fa4127f57b38fc905b9e5b561a73e72676a7320c6e33fb82a0ca19d784242" gracePeriod=30
Oct 30 05:14:41.751064 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:14:41.751020    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:14:43.972509 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:43.972475    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"
Oct 30 05:14:44.067097 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.067038    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ff9e15-586c-4b92-9e47-7d79149ebe93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ff9e15-586c-4b92-9e47-7d79149ebe93-plugins-config-volume\") pod \"f8ff9e15-586c-4b92-9e47-7d79149ebe93\" (UID: \"f8ff9e15-586c-4b92-9e47-7d79149ebe93\") "
Oct 30 05:14:44.067310 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.067151    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ff9e15-586c-4b92-9e47-7d79149ebe93-kube-api-access-4x455\" (UniqueName: \"kubernetes.io/projected/f8ff9e15-586c-4b92-9e47-7d79149ebe93-kube-api-access-4x455\") pod \"f8ff9e15-586c-4b92-9e47-7d79149ebe93\" (UID: \"f8ff9e15-586c-4b92-9e47-7d79149ebe93\") "
Oct 30 05:14:44.070330 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.070290    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ff9e15-586c-4b92-9e47-7d79149ebe93-plugins-config-volume" pod "f8ff9e15-586c-4b92-9e47-7d79149ebe93" (UID: "f8ff9e15-586c-4b92-9e47-7d79149ebe93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:44.071718 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.071685    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ff9e15-586c-4b92-9e47-7d79149ebe93-kube-api-access-4x455" pod "f8ff9e15-586c-4b92-9e47-7d79149ebe93" (UID: "f8ff9e15-586c-4b92-9e47-7d79149ebe93"). InnerVolumeSpecName "kube-api-access-4x455". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:44.168777 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.168743    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x455\" (UniqueName: \"kubernetes.io/projected/f8ff9e15-586c-4b92-9e47-7d79149ebe93-kube-api-access-4x455\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:44.168777 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.168778    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ff9e15-586c-4b92-9e47-7d79149ebe93-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:44.582409 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.582376    5072 generic.go:356] "Generic (PLEG): container finished" podID="bf9d8933-4567-4682-bbfa-5a459474528b" containerID="2a0fa4127f57b38fc905b9e5b561a73e72676a7320c6e33fb82a0ca19d784242" exitCode=0
Oct 30 05:14:44.582613 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.582451    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h" event={"ID":"bf9d8933-4567-4682-bbfa-5a459474528b","Type":"ContainerDied","Data":"2a0fa4127f57b38fc905b9e5b561a73e72676a7320c6e33fb82a0ca19d784242"}
Oct 30 05:14:44.584013 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.583987    5072 generic.go:356] "Generic (PLEG): container finished" podID="f8ff9e15-586c-4b92-9e47-7d79149ebe93" containerID="f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab" exitCode=0
Oct 30 05:14:44.584162 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.584029    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"
Oct 30 05:14:44.584162 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.584109    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4" event={"ID":"f8ff9e15-586c-4b92-9e47-7d79149ebe93","Type":"ContainerDied","Data":"f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab"}
Oct 30 05:14:44.584162 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.584159    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4" event={"ID":"f8ff9e15-586c-4b92-9e47-7d79149ebe93","Type":"ContainerDied","Data":"4d655e3fc39cdf3c33fb68a90efe8ef2b5f3eb1b9700ca69d01e9b497508fb1d"}
Oct 30 05:14:44.584314 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.584187    5072 scope.go:122] "RemoveContainer" containerID="f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab"
Oct 30 05:14:44.594252 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.594236    5072 scope.go:122] "RemoveContainer" containerID="f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab"
Oct 30 05:14:44.594524 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:14:44.594505    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab\": container with ID starting with f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab not found: ID does not exist" containerID="f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab"
Oct 30 05:14:44.594592 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.594531    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab"} err="failed to get container status \"f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab\": rpc error: code = NotFound desc = could not find container \"f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab\": container with ID starting with f02742c25df8bbb17b830a569cc70d21d8a2c48826440f4743d524029a48b1ab not found: ID does not exist"
Oct 30 05:14:44.639103 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.639042    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"]
Oct 30 05:14:44.642730 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:44.642702    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b195d0-7475f7cb44-wrgf4"]
Oct 30 05:14:45.182253 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.182226    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:14:45.288544 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.288466    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-istio-token\" (UniqueName: \"kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-istio-token\") pod \"bf9d8933-4567-4682-bbfa-5a459474528b\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") "
Oct 30 05:14:45.288544 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.288522    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-kube-api-access-wg75n\" (UniqueName: \"kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-kube-api-access-wg75n\") pod \"bf9d8933-4567-4682-bbfa-5a459474528b\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") "
Oct 30 05:14:45.288799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.288580    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf9d8933-4567-4682-bbfa-5a459474528b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9d8933-4567-4682-bbfa-5a459474528b-istiod-ca-cert\") pod \"bf9d8933-4567-4682-bbfa-5a459474528b\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") "
Oct 30 05:14:45.288799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.288621    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf9d8933-4567-4682-bbfa-5a459474528b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9d8933-4567-4682-bbfa-5a459474528b-istio-podinfo\") pod \"bf9d8933-4567-4682-bbfa-5a459474528b\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") "
Oct 30 05:14:45.288799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.288661    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-certs\") pod \"bf9d8933-4567-4682-bbfa-5a459474528b\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") "
Oct 30 05:14:45.288981 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.288810    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-data\") pod \"bf9d8933-4567-4682-bbfa-5a459474528b\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") "
Oct 30 05:14:45.288981 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.288883    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-socket\") pod \"bf9d8933-4567-4682-bbfa-5a459474528b\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") "
Oct 30 05:14:45.288981 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.288918    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-envoy\") pod \"bf9d8933-4567-4682-bbfa-5a459474528b\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") "
Oct 30 05:14:45.288981 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.288947    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-credential-socket\") pod \"bf9d8933-4567-4682-bbfa-5a459474528b\" (UID: \"bf9d8933-4567-4682-bbfa-5a459474528b\") "
Oct 30 05:14:45.288981 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.288950    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-certs" pod "bf9d8933-4567-4682-bbfa-5a459474528b" (UID: "bf9d8933-4567-4682-bbfa-5a459474528b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.289479 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.289095    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9d8933-4567-4682-bbfa-5a459474528b-istiod-ca-cert" pod "bf9d8933-4567-4682-bbfa-5a459474528b" (UID: "bf9d8933-4567-4682-bbfa-5a459474528b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:45.289479 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.289204    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-credential-socket" pod "bf9d8933-4567-4682-bbfa-5a459474528b" (UID: "bf9d8933-4567-4682-bbfa-5a459474528b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.289479 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.289418    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-data" pod "bf9d8933-4567-4682-bbfa-5a459474528b" (UID: "bf9d8933-4567-4682-bbfa-5a459474528b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.289668 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.289479    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-socket" pod "bf9d8933-4567-4682-bbfa-5a459474528b" (UID: "bf9d8933-4567-4682-bbfa-5a459474528b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.289786 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.289762    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.289853 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.289794    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.289853 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.289810    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.289853 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.289827    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9d8933-4567-4682-bbfa-5a459474528b-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.289853 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.289842    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.290866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.290844    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-istio-token" pod "bf9d8933-4567-4682-bbfa-5a459474528b" (UID: "bf9d8933-4567-4682-bbfa-5a459474528b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:45.290998 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.290981    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf9d8933-4567-4682-bbfa-5a459474528b-istio-podinfo" pod "bf9d8933-4567-4682-bbfa-5a459474528b" (UID: "bf9d8933-4567-4682-bbfa-5a459474528b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:14:45.291070 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.291053    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-kube-api-access-wg75n" pod "bf9d8933-4567-4682-bbfa-5a459474528b" (UID: "bf9d8933-4567-4682-bbfa-5a459474528b"). InnerVolumeSpecName "kube-api-access-wg75n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:45.291309 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.291293    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-envoy" pod "bf9d8933-4567-4682-bbfa-5a459474528b" (UID: "bf9d8933-4567-4682-bbfa-5a459474528b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.390721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.390689    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9d8933-4567-4682-bbfa-5a459474528b-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.390721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.390718    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9d8933-4567-4682-bbfa-5a459474528b-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.390883 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.390730    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.390883 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.390742    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg75n\" (UniqueName: \"kubernetes.io/projected/bf9d8933-4567-4682-bbfa-5a459474528b-kube-api-access-wg75n\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.592945 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.592637    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"
Oct 30 05:14:45.592945 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.592636    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h" event={"ID":"bf9d8933-4567-4682-bbfa-5a459474528b","Type":"ContainerDied","Data":"5cee50b8344942f573ca144ed0001cb18a67313fe7834c798d19f2fa442602ca"}
Oct 30 05:14:45.592945 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.592769    5072 scope.go:122] "RemoveContainer" containerID="2a0fa4127f57b38fc905b9e5b561a73e72676a7320c6e33fb82a0ca19d784242"
Oct 30 05:14:45.648710 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.648683    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"]
Oct 30 05:14:45.661431 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:45.661407    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-lm72h"]
Oct 30 05:14:46.303298 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:46.303262    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"]
Oct 30 05:14:46.417096 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:46.409436    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf9d8933-4567-4682-bbfa-5a459474528b" path="/var/lib/kubelet/pods/bf9d8933-4567-4682-bbfa-5a459474528b/volumes"
Oct 30 05:14:46.417096 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:46.410213    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ff9e15-586c-4b92-9e47-7d79149ebe93" path="/var/lib/kubelet/pods/f8ff9e15-586c-4b92-9e47-7d79149ebe93/volumes"
Oct 30 05:14:51.304040 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:51.303974    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t" podUID="ed107f59-f9a9-4a99-8a89-26a288611dd1" containerName="payload-pre-processing" containerID="cri-o://2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3" gracePeriod=25
Oct 30 05:14:52.362876 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:52.362841    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t" podUID="ed107f59-f9a9-4a99-8a89-26a288611dd1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:56.826139 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:56.826099    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"]
Oct 30 05:14:56.856452 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:14:56.856411    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"]
Oct 30 05:15:00.162492 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.162457    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"]
Oct 30 05:15:00.163791 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.163764    5072 cpu_manager.go:414] "containerMap: removing container" podUID="bf5d06ae-de18-4fef-8538-477be6f43278" containerName="payload-pre-processing"
Oct 30 05:15:00.163940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.163927    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5d06ae-de18-4fef-8538-477be6f43278" containerName="payload-pre-processing"
Oct 30 05:15:00.164029 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164019    5072 cpu_manager.go:414] "containerMap: removing container" podUID="93480c6a-fe04-426b-9ee4-cdfd827f2ab9" containerName="payload-processing"
Oct 30 05:15:00.164131 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164120    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93480c6a-fe04-426b-9ee4-cdfd827f2ab9" containerName="payload-processing"
Oct 30 05:15:00.164219 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164210    5072 cpu_manager.go:414] "containerMap: removing container" podUID="23dd655b-f213-4e21-8586-7413a7bea249" containerName="payload-pre-processing"
Oct 30 05:15:00.164287 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164278    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23dd655b-f213-4e21-8586-7413a7bea249" containerName="payload-pre-processing"
Oct 30 05:15:00.164366 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164357    5072 cpu_manager.go:414] "containerMap: removing container" podUID="b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a" containerName="payload-processing"
Oct 30 05:15:00.164442 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164432    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a" containerName="payload-processing"
Oct 30 05:15:00.164523 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164514    5072 cpu_manager.go:414] "containerMap: removing container" podUID="1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0" containerName="payload-pre-processing"
Oct 30 05:15:00.164593 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164585    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0" containerName="payload-pre-processing"
Oct 30 05:15:00.164677 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164668    5072 cpu_manager.go:414] "containerMap: removing container" podUID="75405bbd-d694-4d30-9c5b-122843efcd55" containerName="payload-processing"
Oct 30 05:15:00.164747 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164738    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75405bbd-d694-4d30-9c5b-122843efcd55" containerName="payload-processing"
Oct 30 05:15:00.164815 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164807    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f8ff9e15-586c-4b92-9e47-7d79149ebe93" containerName="payload-processing"
Oct 30 05:15:00.164887 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164877    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ff9e15-586c-4b92-9e47-7d79149ebe93" containerName="payload-processing"
Oct 30 05:15:00.164969 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.164961    5072 cpu_manager.go:414] "containerMap: removing container" podUID="bf9d8933-4567-4682-bbfa-5a459474528b" containerName="istio-proxy"
Oct 30 05:15:00.165036 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.165027    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9d8933-4567-4682-bbfa-5a459474528b" containerName="istio-proxy"
Oct 30 05:15:00.165129 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.165119    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7366ae8d-b60d-42bf-942f-16f3c2356fef" containerName="revoke-keys"
Oct 30 05:15:00.165200 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.165191    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7366ae8d-b60d-42bf-942f-16f3c2356fef" containerName="revoke-keys"
Oct 30 05:15:00.165273 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.165265    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e7105916-eb76-4348-86d3-820f79627195" containerName="main"
Oct 30 05:15:00.165340 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.165332    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7105916-eb76-4348-86d3-820f79627195" containerName="main"
Oct 30 05:15:00.165407 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.165399    5072 cpu_manager.go:414] "containerMap: removing container" podUID="3a053e0d-687d-4926-b1d5-cb40e2c769f9" containerName="main"
Oct 30 05:15:00.165478 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.165469    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a053e0d-687d-4926-b1d5-cb40e2c769f9" containerName="main"
Oct 30 05:15:00.165689 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.165678    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="93480c6a-fe04-426b-9ee4-cdfd827f2ab9" containerName="payload-processing"
Oct 30 05:15:00.165767 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.165759    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="23dd655b-f213-4e21-8586-7413a7bea249" containerName="payload-pre-processing"
Oct 30 05:15:00.165844 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.165836    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f01ab3-09fc-4710-b182-a3d2d3dc4f2a" containerName="payload-processing"
Oct 30 05:15:00.165913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.165905    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec101f2-4aa8-4a85-ab2b-a30f9fe298f0" containerName="payload-pre-processing"
Oct 30 05:15:00.165982 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.165974    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="75405bbd-d694-4d30-9c5b-122843efcd55" containerName="payload-processing"
Oct 30 05:15:00.166053 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.166045    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ff9e15-586c-4b92-9e47-7d79149ebe93" containerName="payload-processing"
Oct 30 05:15:00.166149 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.166138    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9d8933-4567-4682-bbfa-5a459474528b" containerName="istio-proxy"
Oct 30 05:15:00.166227 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.166219    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7366ae8d-b60d-42bf-942f-16f3c2356fef" containerName="revoke-keys"
Oct 30 05:15:00.166290 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.166283    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7105916-eb76-4348-86d3-820f79627195" containerName="main"
Oct 30 05:15:00.166365 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.166357    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a053e0d-687d-4926-b1d5-cb40e2c769f9" containerName="main"
Oct 30 05:15:00.166440 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.166432    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5d06ae-de18-4fef-8538-477be6f43278" containerName="payload-pre-processing"
Oct 30 05:15:00.179990 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.179964    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"
Oct 30 05:15:00.190440 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.190412    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"]
Oct 30 05:15:00.197827 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.197804    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:15:00.257296 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.257257    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6z4\" (UniqueName: \"kubernetes.io/projected/628d4322-92fa-4b80-9314-b2c8ca202561-kube-api-access-7c6z4\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj\" (UID: \"628d4322-92fa-4b80-9314-b2c8ca202561\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"
Oct 30 05:15:00.257481 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.257313    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628d4322-92fa-4b80-9314-b2c8ca202561-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj\" (UID: \"628d4322-92fa-4b80-9314-b2c8ca202561\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"
Oct 30 05:15:00.359093 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.359051    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6z4\" (UniqueName: \"kubernetes.io/projected/628d4322-92fa-4b80-9314-b2c8ca202561-kube-api-access-7c6z4\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj\" (UID: \"628d4322-92fa-4b80-9314-b2c8ca202561\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"
Oct 30 05:15:00.359279 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.359126    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628d4322-92fa-4b80-9314-b2c8ca202561-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj\" (UID: \"628d4322-92fa-4b80-9314-b2c8ca202561\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"
Oct 30 05:15:00.359800 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.359775    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628d4322-92fa-4b80-9314-b2c8ca202561-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj\" (UID: \"628d4322-92fa-4b80-9314-b2c8ca202561\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"
Oct 30 05:15:00.383292 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.383255    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6z4\" (UniqueName: \"kubernetes.io/projected/628d4322-92fa-4b80-9314-b2c8ca202561-kube-api-access-7c6z4\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj\" (UID: \"628d4322-92fa-4b80-9314-b2c8ca202561\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"
Oct 30 05:15:00.519724 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.519698    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:15:00.525831 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.525812    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"
Oct 30 05:15:00.834641 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.834610    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"]
Oct 30 05:15:00.834761 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:15:00.834667    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628d4322_92fa_4b80_9314_b2c8ca202561.slice/crio-49514306b7dbe9ec6a7bbd6241b1a326d59e85a45dcc5d582a674bdd6d883524 WatchSource:0}: Error finding container 49514306b7dbe9ec6a7bbd6241b1a326d59e85a45dcc5d582a674bdd6d883524: Status 404 returned error can't find the container with id 49514306b7dbe9ec6a7bbd6241b1a326d59e85a45dcc5d582a674bdd6d883524
Oct 30 05:15:00.903520 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.903485    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"]
Oct 30 05:15:00.903688 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:00.903672    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk" podUID="4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" containerName="istio-proxy" containerID="cri-o://3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6" gracePeriod=30
Oct 30 05:15:01.700481 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:01.700440    5072 generic.go:356] "Generic (PLEG): container finished" podID="628d4322-92fa-4b80-9314-b2c8ca202561" containerID="4535d19f4dcab72735d69b4f246b0b7d3e12e6c6d72fab68b9627c95f0911b3a" exitCode=60
Oct 30 05:15:01.700930 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:01.700531    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj" event={"ID":"628d4322-92fa-4b80-9314-b2c8ca202561","Type":"ContainerDied","Data":"4535d19f4dcab72735d69b4f246b0b7d3e12e6c6d72fab68b9627c95f0911b3a"}
Oct 30 05:15:01.700930 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:01.700626    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj" event={"ID":"628d4322-92fa-4b80-9314-b2c8ca202561","Type":"ContainerStarted","Data":"49514306b7dbe9ec6a7bbd6241b1a326d59e85a45dcc5d582a674bdd6d883524"}
Oct 30 05:15:01.700930 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:01.700815    5072 scope.go:122] "RemoveContainer" containerID="4535d19f4dcab72735d69b4f246b0b7d3e12e6c6d72fab68b9627c95f0911b3a"
Oct 30 05:15:01.826428 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:01.826386    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m" podUID="c260ccd2-e290-4e56-84f0-2ebc79a5fcf7" containerName="payload-processing" containerID="cri-o://cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9" gracePeriod=25
Oct 30 05:15:01.856688 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:01.856643    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl" podUID="35d1ddbc-3156-44bf-be1d-9f236418a2b7" containerName="payload-pre-processing" containerID="cri-o://b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb" gracePeriod=25
Oct 30 05:15:01.959022 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:01.958996    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"]
Oct 30 05:15:01.964905 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:01.964883    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:02.003962 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.003928    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"]
Oct 30 05:15:02.011005 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.010977    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:15:02.078465 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.078420    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmw6l\" (UniqueName: \"kubernetes.io/projected/b2bf0a67-0a7e-46cf-a27c-2789223e072e-kube-api-access-mmw6l\") pod \"maas-api-revoke-sub-b68edfb1-vqnrk\" (UID: \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:02.078653 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.078537    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b68edfb1-vqnrk\" (UID: \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:02.078653 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.078609    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b68edfb1-vqnrk\" (UID: \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:02.179947 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.179914    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b68edfb1-vqnrk\" (UID: \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:02.180138 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.180002    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b68edfb1-vqnrk\" (UID: \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:02.180138 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.180064    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmw6l\" (UniqueName: \"kubernetes.io/projected/b2bf0a67-0a7e-46cf-a27c-2789223e072e-kube-api-access-mmw6l\") pod \"maas-api-revoke-sub-b68edfb1-vqnrk\" (UID: \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:02.180623 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.180600    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b68edfb1-vqnrk\" (UID: \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:02.182586 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.182536    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b68edfb1-vqnrk\" (UID: \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:02.203888 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.203862    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmw6l\" (UniqueName: \"kubernetes.io/projected/b2bf0a67-0a7e-46cf-a27c-2789223e072e-kube-api-access-mmw6l\") pod \"maas-api-revoke-sub-b68edfb1-vqnrk\" (UID: \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:02.280050 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.279954    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:02.363885 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.363848    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t" podUID="ed107f59-f9a9-4a99-8a89-26a288611dd1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:15:02.588575 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:15:02.588545    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bf0a67_0a7e_46cf_a27c_2789223e072e.slice/crio-3ee6657c3b651dfdc6d0c512fca651c3d8d56d351efec33171927ce0094860dd WatchSource:0}: Error finding container 3ee6657c3b651dfdc6d0c512fca651c3d8d56d351efec33171927ce0094860dd: Status 404 returned error can't find the container with id 3ee6657c3b651dfdc6d0c512fca651c3d8d56d351efec33171927ce0094860dd
Oct 30 05:15:02.589563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.589538    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"]
Oct 30 05:15:02.709661 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.709630    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk" event={"ID":"b2bf0a67-0a7e-46cf-a27c-2789223e072e","Type":"ContainerStarted","Data":"3ee6657c3b651dfdc6d0c512fca651c3d8d56d351efec33171927ce0094860dd"}
Oct 30 05:15:02.711665 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.711637    5072 generic.go:356] "Generic (PLEG): container finished" podID="628d4322-92fa-4b80-9314-b2c8ca202561" containerID="4ec38ae36ee53b02e831ea85557d84e5051f993424580d7e6160da5ed8198fe6" exitCode=60
Oct 30 05:15:02.711784 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.711695    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj" event={"ID":"628d4322-92fa-4b80-9314-b2c8ca202561","Type":"ContainerDied","Data":"4ec38ae36ee53b02e831ea85557d84e5051f993424580d7e6160da5ed8198fe6"}
Oct 30 05:15:02.711784 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.711745    5072 scope.go:122] "RemoveContainer" containerID="4535d19f4dcab72735d69b4f246b0b7d3e12e6c6d72fab68b9627c95f0911b3a"
Oct 30 05:15:02.712024 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:02.712002    5072 scope.go:122] "RemoveContainer" containerID="4ec38ae36ee53b02e831ea85557d84e5051f993424580d7e6160da5ed8198fe6"
Oct 30 05:15:02.712367 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:02.712335    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj_odh-ai-gateway-infra(628d4322-92fa-4b80-9314-b2c8ca202561)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj" podUID="628d4322-92fa-4b80-9314-b2c8ca202561"
Oct 30 05:15:03.720683 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:03.720645    5072 generic.go:356] "Generic (PLEG): container finished" podID="b2bf0a67-0a7e-46cf-a27c-2789223e072e" containerID="8be449d695e36aa15f1c5afd4e708e51a54d32f78922781d4200595e005f1b0b" exitCode=0
Oct 30 05:15:03.721155 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:03.720733    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk" event={"ID":"b2bf0a67-0a7e-46cf-a27c-2789223e072e","Type":"ContainerDied","Data":"8be449d695e36aa15f1c5afd4e708e51a54d32f78922781d4200595e005f1b0b"}
Oct 30 05:15:03.722568 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:03.722551    5072 scope.go:122] "RemoveContainer" containerID="4ec38ae36ee53b02e831ea85557d84e5051f993424580d7e6160da5ed8198fe6"
Oct 30 05:15:03.722787 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:03.722769    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj_odh-ai-gateway-infra(628d4322-92fa-4b80-9314-b2c8ca202561)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj" podUID="628d4322-92fa-4b80-9314-b2c8ca202561"
Oct 30 05:15:04.942718 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:04.942692    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:05.012860 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:05.012777    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-xks-ca\") pod \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\" (UID: \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\") "
Oct 30 05:15:05.012860 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:05.012825    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bf0a67-0a7e-46cf-a27c-2789223e072e-kube-api-access-mmw6l\" (UniqueName: \"kubernetes.io/projected/b2bf0a67-0a7e-46cf-a27c-2789223e072e-kube-api-access-mmw6l\") pod \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\" (UID: \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\") "
Oct 30 05:15:05.013121 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:05.013000    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-openshift-ca\") pod \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\" (UID: \"b2bf0a67-0a7e-46cf-a27c-2789223e072e\") "
Oct 30 05:15:05.013388 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:05.013364    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-openshift-ca" pod "b2bf0a67-0a7e-46cf-a27c-2789223e072e" (UID: "b2bf0a67-0a7e-46cf-a27c-2789223e072e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:05.013483 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:05.013463    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:05.015690 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:05.015666    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-xks-ca" pod "b2bf0a67-0a7e-46cf-a27c-2789223e072e" (UID: "b2bf0a67-0a7e-46cf-a27c-2789223e072e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:15:05.015769 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:05.015693    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bf0a67-0a7e-46cf-a27c-2789223e072e-kube-api-access-mmw6l" pod "b2bf0a67-0a7e-46cf-a27c-2789223e072e" (UID: "b2bf0a67-0a7e-46cf-a27c-2789223e072e"). InnerVolumeSpecName "kube-api-access-mmw6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:05.114695 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:05.114666    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bf0a67-0a7e-46cf-a27c-2789223e072e-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:05.114695 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:05.114695    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmw6l\" (UniqueName: \"kubernetes.io/projected/b2bf0a67-0a7e-46cf-a27c-2789223e072e-kube-api-access-mmw6l\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:05.738511 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:05.738486    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"
Oct 30 05:15:05.738679 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:05.738484    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk" event={"ID":"b2bf0a67-0a7e-46cf-a27c-2789223e072e","Type":"ContainerDied","Data":"3ee6657c3b651dfdc6d0c512fca651c3d8d56d351efec33171927ce0094860dd"}
Oct 30 05:15:05.738679 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:05.738609    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ee6657c3b651dfdc6d0c512fca651c3d8d56d351efec33171927ce0094860dd"
Oct 30 05:15:06.179259 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.179233    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:15:06.329695 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.329626    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-envoy\") pod \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") "
Oct 30 05:15:06.329695 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.329661    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-data\") pod \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") "
Oct 30 05:15:06.329899 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.329720    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-certs\") pod \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") "
Oct 30 05:15:06.329899 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.329754    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-token\" (UniqueName: \"kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-token\") pod \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") "
Oct 30 05:15:06.329899 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.329788    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-kube-api-access-zbq89\" (UniqueName: \"kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-kube-api-access-zbq89\") pod \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") "
Oct 30 05:15:06.329899 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.329830    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-podinfo\") pod \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") "
Oct 30 05:15:06.330143 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.329901    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-credential-socket\") pod \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") "
Oct 30 05:15:06.330143 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.329955    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-data" pod "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" (UID: "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:06.330143 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.330046    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istiod-ca-cert\") pod \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") "
Oct 30 05:15:06.330143 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.330047    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-certs" pod "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" (UID: "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:06.330143 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.330106    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-socket\") pod \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\" (UID: \"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce\") "
Oct 30 05:15:06.330433 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.330263    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-credential-socket" pod "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" (UID: "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:06.330494 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.330443    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-socket" pod "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" (UID: "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:06.330494 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.330475    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istiod-ca-cert" pod "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" (UID: "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:06.330728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.330588    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.330728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.330605    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.330728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.330619    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.330728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.330632    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.330728 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.330646    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.332401 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.332373    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-envoy" pod "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" (UID: "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:06.332549 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.332457    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-podinfo" pod "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" (UID: "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:15:06.332742 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.332715    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-token" pod "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" (UID: "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:06.332742 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.332729    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-kube-api-access-zbq89" pod "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" (UID: "4dbdae54-cfe9-4ced-a212-2cfc4e1372ce"). InnerVolumeSpecName "kube-api-access-zbq89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:06.429472 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.429444    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"]
Oct 30 05:15:06.431732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.431673    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.431732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.431701    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.431732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.431711    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbq89\" (UniqueName: \"kubernetes.io/projected/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-kube-api-access-zbq89\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.431732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.431720    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.748248 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.748197    5072 generic.go:356] "Generic (PLEG): container finished" podID="4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" containerID="3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6" exitCode=0
Oct 30 05:15:06.748248 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.748248    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"
Oct 30 05:15:06.748544 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.748280    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk" event={"ID":"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce","Type":"ContainerDied","Data":"3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6"}
Oct 30 05:15:06.748544 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.748337    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk" event={"ID":"4dbdae54-cfe9-4ced-a212-2cfc4e1372ce","Type":"ContainerDied","Data":"6f637e95c503e2f977d14b64c0394d4cf114750b49521595d9b676ec3040e5af"}
Oct 30 05:15:06.748544 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.748364    5072 scope.go:122] "RemoveContainer" containerID="3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6"
Oct 30 05:15:06.759595 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.759573    5072 scope.go:122] "RemoveContainer" containerID="3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6"
Oct 30 05:15:06.759973 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:06.759945    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6\": container with ID starting with 3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6 not found: ID does not exist" containerID="3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6"
Oct 30 05:15:06.760135 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.759983    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6"} err="failed to get container status \"3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6\": rpc error: code = NotFound desc = could not find container \"3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6\": container with ID starting with 3cde84f224d52ad32a926e853aee30ddbdbaa7c1696bd452d92024d87d1aece6 not found: ID does not exist"
Oct 30 05:15:06.804345 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.804319    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"]
Oct 30 05:15:06.817229 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:06.817199    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-qjdwk"]
Oct 30 05:15:08.262498 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:08.262449    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl" podUID="35d1ddbc-3156-44bf-be1d-9f236418a2b7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:15:08.263035 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:08.262743    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m" podUID="c260ccd2-e290-4e56-84f0-2ebc79a5fcf7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:15:08.407996 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:08.407966    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" path="/var/lib/kubelet/pods/4dbdae54-cfe9-4ced-a212-2cfc4e1372ce/volumes"
Oct 30 05:15:10.256511 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:10.256473    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"]
Oct 30 05:15:11.581268 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.581152    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"
Oct 30 05:15:11.696085 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.696052    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed107f59-f9a9-4a99-8a89-26a288611dd1-kube-api-access-8zdqh\" (UniqueName: \"kubernetes.io/projected/ed107f59-f9a9-4a99-8a89-26a288611dd1-kube-api-access-8zdqh\") pod \"ed107f59-f9a9-4a99-8a89-26a288611dd1\" (UID: \"ed107f59-f9a9-4a99-8a89-26a288611dd1\") "
Oct 30 05:15:11.696258 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.696168    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed107f59-f9a9-4a99-8a89-26a288611dd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed107f59-f9a9-4a99-8a89-26a288611dd1-plugins-config-volume\") pod \"ed107f59-f9a9-4a99-8a89-26a288611dd1\" (UID: \"ed107f59-f9a9-4a99-8a89-26a288611dd1\") "
Oct 30 05:15:11.696617 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.696592    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed107f59-f9a9-4a99-8a89-26a288611dd1-plugins-config-volume" pod "ed107f59-f9a9-4a99-8a89-26a288611dd1" (UID: "ed107f59-f9a9-4a99-8a89-26a288611dd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:11.698378 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.698356    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed107f59-f9a9-4a99-8a89-26a288611dd1-kube-api-access-8zdqh" pod "ed107f59-f9a9-4a99-8a89-26a288611dd1" (UID: "ed107f59-f9a9-4a99-8a89-26a288611dd1"). InnerVolumeSpecName "kube-api-access-8zdqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:11.752221 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:11.752181    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:15:11.783567 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.783536    5072 generic.go:356] "Generic (PLEG): container finished" podID="ed107f59-f9a9-4a99-8a89-26a288611dd1" containerID="2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3" exitCode=0
Oct 30 05:15:11.783770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.783579    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"
Oct 30 05:15:11.783770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.783620    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t" event={"ID":"ed107f59-f9a9-4a99-8a89-26a288611dd1","Type":"ContainerDied","Data":"2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3"}
Oct 30 05:15:11.783770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.783699    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t" event={"ID":"ed107f59-f9a9-4a99-8a89-26a288611dd1","Type":"ContainerDied","Data":"36ab5f0f1c97bf0ab97f14f9c8abb8c8be8e75f8c5d388a598c98cca47b1b2ba"}
Oct 30 05:15:11.783770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.783725    5072 scope.go:122] "RemoveContainer" containerID="2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3"
Oct 30 05:15:11.794512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.794495    5072 scope.go:122] "RemoveContainer" containerID="2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3"
Oct 30 05:15:11.794779 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:11.794760    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3\": container with ID starting with 2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3 not found: ID does not exist" containerID="2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3"
Oct 30 05:15:11.794842 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.794786    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3"} err="failed to get container status \"2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3\": rpc error: code = NotFound desc = could not find container \"2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3\": container with ID starting with 2a09236be4778b153ec49d439d8bf848157dcb1449cfed4f61709a5ec4619ed3 not found: ID does not exist"
Oct 30 05:15:11.797411 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.797393    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdqh\" (UniqueName: \"kubernetes.io/projected/ed107f59-f9a9-4a99-8a89-26a288611dd1-kube-api-access-8zdqh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:11.797467 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.797414    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed107f59-f9a9-4a99-8a89-26a288611dd1-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:11.838794 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.838733    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"]
Oct 30 05:15:11.843419 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:11.843387    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-247158-7b5588b64c-29z2t"]
Oct 30 05:15:12.408447 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:12.408404    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed107f59-f9a9-4a99-8a89-26a288611dd1" path="/var/lib/kubelet/pods/ed107f59-f9a9-4a99-8a89-26a288611dd1/volumes"
Oct 30 05:15:14.028367 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:14.028331    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"]
Oct 30 05:15:14.042205 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:14.042172    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f010de4b-hjr5z"]
Oct 30 05:15:14.400535 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:14.400459    5072 scope.go:122] "RemoveContainer" containerID="4ec38ae36ee53b02e831ea85557d84e5051f993424580d7e6160da5ed8198fe6"
Oct 30 05:15:14.408528 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:14.408506    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc76efb-e407-4e83-b4cb-ca9a947def75" path="/var/lib/kubelet/pods/0bc76efb-e407-4e83-b4cb-ca9a947def75/volumes"
Oct 30 05:15:14.813428 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:14.813398    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj" event={"ID":"628d4322-92fa-4b80-9314-b2c8ca202561","Type":"ContainerStarted","Data":"eca06f9f3140135c3ef4b3158fb197e57a05d019d9f6d38cd8ed435f466b790a"}
Oct 30 05:15:14.856491 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:14.856412    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj" podStartSLOduration=14.856397311 podStartE2EDuration="14.856397311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:14.85415376 +0000 UTC m=+4694.595576857" watchObservedRunningTime="2026-10-30 05:15:14.856397311 +0000 UTC m=+4694.597820408"
Oct 30 05:15:15.821632 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:15.821599    5072 generic.go:356] "Generic (PLEG): container finished" podID="628d4322-92fa-4b80-9314-b2c8ca202561" containerID="eca06f9f3140135c3ef4b3158fb197e57a05d019d9f6d38cd8ed435f466b790a" exitCode=60
Oct 30 05:15:15.822142 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:15.821673    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj" event={"ID":"628d4322-92fa-4b80-9314-b2c8ca202561","Type":"ContainerDied","Data":"eca06f9f3140135c3ef4b3158fb197e57a05d019d9f6d38cd8ed435f466b790a"}
Oct 30 05:15:15.822142 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:15.821731    5072 scope.go:122] "RemoveContainer" containerID="4ec38ae36ee53b02e831ea85557d84e5051f993424580d7e6160da5ed8198fe6"
Oct 30 05:15:15.822286 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:15.822156    5072 scope.go:122] "RemoveContainer" containerID="eca06f9f3140135c3ef4b3158fb197e57a05d019d9f6d38cd8ed435f466b790a"
Oct 30 05:15:15.822480 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:15.822458    5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj_odh-ai-gateway-infra(628d4322-92fa-4b80-9314-b2c8ca202561)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj" podUID="628d4322-92fa-4b80-9314-b2c8ca202561"
Oct 30 05:15:15.865375 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:15.865351    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"]
Oct 30 05:15:16.985467 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:16.985439    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"
Oct 30 05:15:17.058614 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:17.058586    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/628d4322-92fa-4b80-9314-b2c8ca202561-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628d4322-92fa-4b80-9314-b2c8ca202561-maas-api-service-ca\") pod \"628d4322-92fa-4b80-9314-b2c8ca202561\" (UID: \"628d4322-92fa-4b80-9314-b2c8ca202561\") "
Oct 30 05:15:17.058804 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:17.058710    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/628d4322-92fa-4b80-9314-b2c8ca202561-kube-api-access-7c6z4\" (UniqueName: \"kubernetes.io/projected/628d4322-92fa-4b80-9314-b2c8ca202561-kube-api-access-7c6z4\") pod \"628d4322-92fa-4b80-9314-b2c8ca202561\" (UID: \"628d4322-92fa-4b80-9314-b2c8ca202561\") "
Oct 30 05:15:17.058985 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:17.058962    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628d4322-92fa-4b80-9314-b2c8ca202561-maas-api-service-ca" pod "628d4322-92fa-4b80-9314-b2c8ca202561" (UID: "628d4322-92fa-4b80-9314-b2c8ca202561"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:17.059211 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:17.059192    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628d4322-92fa-4b80-9314-b2c8ca202561-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:17.061234 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:17.061206    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628d4322-92fa-4b80-9314-b2c8ca202561-kube-api-access-7c6z4" pod "628d4322-92fa-4b80-9314-b2c8ca202561" (UID: "628d4322-92fa-4b80-9314-b2c8ca202561"). InnerVolumeSpecName "kube-api-access-7c6z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:17.160711 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:17.160630    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c6z4\" (UniqueName: \"kubernetes.io/projected/628d4322-92fa-4b80-9314-b2c8ca202561-kube-api-access-7c6z4\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:17.838820 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:17.838778    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"
Oct 30 05:15:17.838995 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:17.838808    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj" event={"ID":"628d4322-92fa-4b80-9314-b2c8ca202561","Type":"ContainerDied","Data":"49514306b7dbe9ec6a7bbd6241b1a326d59e85a45dcc5d582a674bdd6d883524"}
Oct 30 05:15:17.838995 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:17.838892    5072 scope.go:122] "RemoveContainer" containerID="eca06f9f3140135c3ef4b3158fb197e57a05d019d9f6d38cd8ed435f466b790a"
Oct 30 05:15:17.896412 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:17.896377    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"]
Oct 30 05:15:17.909489 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:17.909464    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888955-bzwfj"]
Oct 30 05:15:18.262118 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:18.262045    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl" podUID="35d1ddbc-3156-44bf-be1d-9f236418a2b7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:15:18.262531 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:18.262149    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m" podUID="c260ccd2-e290-4e56-84f0-2ebc79a5fcf7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:15:18.408136 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:18.408106    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="628d4322-92fa-4b80-9314-b2c8ca202561" path="/var/lib/kubelet/pods/628d4322-92fa-4b80-9314-b2c8ca202561/volumes"
Oct 30 05:15:21.452177 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.452138    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8" podUID="dc6cf57f-0ddb-4e76-a06b-1cddd5e24070" containerName="main" containerID="cri-o://482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591" gracePeriod=45
Oct 30 05:15:21.706499 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.706435    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 05:15:21.808863 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.808824    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-dshm\") pod \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") "
Oct 30 05:15:21.809065 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.808872    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-kube-api-access-qtcqb\" (UniqueName: \"kubernetes.io/projected/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-kube-api-access-qtcqb\") pod \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") "
Oct 30 05:15:21.809065 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.808908    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tls-certs\") pod \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") "
Oct 30 05:15:21.809065 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.808969    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-home\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-home\") pod \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") "
Oct 30 05:15:21.809307 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.809065    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-model-cache\") pod \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") "
Oct 30 05:15:21.809307 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.809166    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tmp-dir\") pod \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\" (UID: \"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070\") "
Oct 30 05:15:21.809683 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.809620    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-model-cache" pod "dc6cf57f-0ddb-4e76-a06b-1cddd5e24070" (UID: "dc6cf57f-0ddb-4e76-a06b-1cddd5e24070"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:21.809778 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.809674    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-home" pod "dc6cf57f-0ddb-4e76-a06b-1cddd5e24070" (UID: "dc6cf57f-0ddb-4e76-a06b-1cddd5e24070"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:21.810119 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.810090    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tmp-dir" pod "dc6cf57f-0ddb-4e76-a06b-1cddd5e24070" (UID: "dc6cf57f-0ddb-4e76-a06b-1cddd5e24070"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:21.811424 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.811398    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-dshm" pod "dc6cf57f-0ddb-4e76-a06b-1cddd5e24070" (UID: "dc6cf57f-0ddb-4e76-a06b-1cddd5e24070"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:21.811771 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.811747    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tls-certs" pod "dc6cf57f-0ddb-4e76-a06b-1cddd5e24070" (UID: "dc6cf57f-0ddb-4e76-a06b-1cddd5e24070"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:15:21.811946 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.811926    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-kube-api-access-qtcqb" pod "dc6cf57f-0ddb-4e76-a06b-1cddd5e24070" (UID: "dc6cf57f-0ddb-4e76-a06b-1cddd5e24070"). InnerVolumeSpecName "kube-api-access-qtcqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:21.873926 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.873890    5072 generic.go:356] "Generic (PLEG): container finished" podID="dc6cf57f-0ddb-4e76-a06b-1cddd5e24070" containerID="482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591" exitCode=0
Oct 30 05:15:21.874067 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.873966    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8" event={"ID":"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070","Type":"ContainerDied","Data":"482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591"}
Oct 30 05:15:21.874067 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.874001    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8" event={"ID":"dc6cf57f-0ddb-4e76-a06b-1cddd5e24070","Type":"ContainerDied","Data":"93903597b64e5d9dadb7c0144456ecbd9888ae1dddb21b3fc4a1301756402d23"}
Oct 30 05:15:21.874067 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.874028    5072 scope.go:122] "RemoveContainer" containerID="482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591"
Oct 30 05:15:21.874282 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.874219    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"
Oct 30 05:15:21.910415 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.910391    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.910514 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.910417    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtcqb\" (UniqueName: \"kubernetes.io/projected/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-kube-api-access-qtcqb\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.910514 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.910427    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.910514 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.910436    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.910514 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.910444    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.910514 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.910468    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.974342 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.974316    5072 scope.go:122] "RemoveContainer" containerID="482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591"
Oct 30 05:15:21.974673 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:21.974643    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591\": container with ID starting with 482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591 not found: ID does not exist" containerID="482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591"
Oct 30 05:15:21.974757 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:21.974686    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591"} err="failed to get container status \"482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591\": rpc error: code = NotFound desc = could not find container \"482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591\": container with ID starting with 482a5c048950bff9e9789ab9f0fb09adb72b867700df19e37405c3dcb1801591 not found: ID does not exist"
Oct 30 05:15:22.004304 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.004274    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"]
Oct 30 05:15:22.012771 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.012738    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/simulator-w0-36bbac-kserve-7584f46d-stfr8"]
Oct 30 05:15:22.110812 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.110785    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"
Oct 30 05:15:22.143309 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.143288    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"
Oct 30 05:15:22.213036 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.213004    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-plugins-config-volume\") pod \"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7\" (UID: \"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7\") "
Oct 30 05:15:22.213228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.213059    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35d1ddbc-3156-44bf-be1d-9f236418a2b7-kube-api-access-4v8sv\" (UniqueName: \"kubernetes.io/projected/35d1ddbc-3156-44bf-be1d-9f236418a2b7-kube-api-access-4v8sv\") pod \"35d1ddbc-3156-44bf-be1d-9f236418a2b7\" (UID: \"35d1ddbc-3156-44bf-be1d-9f236418a2b7\") "
Oct 30 05:15:22.213402 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.213377    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-kube-api-access-c48s5\" (UniqueName: \"kubernetes.io/projected/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-kube-api-access-c48s5\") pod \"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7\" (UID: \"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7\") "
Oct 30 05:15:22.213526 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.213416    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35d1ddbc-3156-44bf-be1d-9f236418a2b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d1ddbc-3156-44bf-be1d-9f236418a2b7-plugins-config-volume\") pod \"35d1ddbc-3156-44bf-be1d-9f236418a2b7\" (UID: \"35d1ddbc-3156-44bf-be1d-9f236418a2b7\") "
Oct 30 05:15:22.213721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.213700    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-plugins-config-volume" pod "c260ccd2-e290-4e56-84f0-2ebc79a5fcf7" (UID: "c260ccd2-e290-4e56-84f0-2ebc79a5fcf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:22.214022 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.213976    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:22.214157 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.214022    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d1ddbc-3156-44bf-be1d-9f236418a2b7-plugins-config-volume" pod "35d1ddbc-3156-44bf-be1d-9f236418a2b7" (UID: "35d1ddbc-3156-44bf-be1d-9f236418a2b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:22.215940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.215887    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-kube-api-access-c48s5" pod "c260ccd2-e290-4e56-84f0-2ebc79a5fcf7" (UID: "c260ccd2-e290-4e56-84f0-2ebc79a5fcf7"). InnerVolumeSpecName "kube-api-access-c48s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:22.216341 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.216317    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d1ddbc-3156-44bf-be1d-9f236418a2b7-kube-api-access-4v8sv" pod "35d1ddbc-3156-44bf-be1d-9f236418a2b7" (UID: "35d1ddbc-3156-44bf-be1d-9f236418a2b7"). InnerVolumeSpecName "kube-api-access-4v8sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:22.314628 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.314599    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v8sv\" (UniqueName: \"kubernetes.io/projected/35d1ddbc-3156-44bf-be1d-9f236418a2b7-kube-api-access-4v8sv\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:22.314628 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.314628    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c48s5\" (UniqueName: \"kubernetes.io/projected/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7-kube-api-access-c48s5\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:22.314628 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.314643    5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d1ddbc-3156-44bf-be1d-9f236418a2b7-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:22.408450 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.408413    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6cf57f-0ddb-4e76-a06b-1cddd5e24070" path="/var/lib/kubelet/pods/dc6cf57f-0ddb-4e76-a06b-1cddd5e24070/volumes"
Oct 30 05:15:22.882412 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.882331    5072 generic.go:356] "Generic (PLEG): container finished" podID="35d1ddbc-3156-44bf-be1d-9f236418a2b7" containerID="b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb" exitCode=0
Oct 30 05:15:22.882412 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.882375    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"
Oct 30 05:15:22.882942 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.882415    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl" event={"ID":"35d1ddbc-3156-44bf-be1d-9f236418a2b7","Type":"ContainerDied","Data":"b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb"}
Oct 30 05:15:22.882942 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.882477    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl" event={"ID":"35d1ddbc-3156-44bf-be1d-9f236418a2b7","Type":"ContainerDied","Data":"8637ff64eb8630c98d6f10d2a4003228e80c6ee05eabb3aeb0c7168c68d11337"}
Oct 30 05:15:22.882942 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.882509    5072 scope.go:122] "RemoveContainer" containerID="b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb"
Oct 30 05:15:22.883864 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.883734    5072 generic.go:356] "Generic (PLEG): container finished" podID="c260ccd2-e290-4e56-84f0-2ebc79a5fcf7" containerID="cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9" exitCode=0
Oct 30 05:15:22.883864 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.883772    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"
Oct 30 05:15:22.883864 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.883808    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m" event={"ID":"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7","Type":"ContainerDied","Data":"cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9"}
Oct 30 05:15:22.883864 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.883856    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m" event={"ID":"c260ccd2-e290-4e56-84f0-2ebc79a5fcf7","Type":"ContainerDied","Data":"5d802c64b464357086c0125dd2b6fe958f2f95e3055b53c3ced58cbe0dc45d32"}
Oct 30 05:15:22.892268 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.892249    5072 scope.go:122] "RemoveContainer" containerID="b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb"
Oct 30 05:15:22.892521 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:22.892502    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb\": container with ID starting with b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb not found: ID does not exist" containerID="b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb"
Oct 30 05:15:22.892603 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.892527    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb"} err="failed to get container status \"b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb\": rpc error: code = NotFound desc = could not find container \"b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb\": container with ID starting with b573e231ec0264216afa99d1fe74545e966698a93eb4d8ae8f0f53b24c9f8bcb not found: ID does not exist"
Oct 30 05:15:22.892603 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.892544    5072 scope.go:122] "RemoveContainer" containerID="cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9"
Oct 30 05:15:22.901635 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.901619    5072 scope.go:122] "RemoveContainer" containerID="cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9"
Oct 30 05:15:22.901874 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:22.901855    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9\": container with ID starting with cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9 not found: ID does not exist" containerID="cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9"
Oct 30 05:15:22.901947 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.901880    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9"} err="failed to get container status \"cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9\": rpc error: code = NotFound desc = could not find container \"cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9\": container with ID starting with cfaf54a5693298ce5a9980289e488667f6754bcb28e560764e55681180aba6a9 not found: ID does not exist"
Oct 30 05:15:22.937686 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.937663    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"]
Oct 30 05:15:22.950604 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.950582    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-495ebe-d9cdfd7ff-6fznl"]
Oct 30 05:15:22.984711 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.984690    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"]
Oct 30 05:15:22.988282 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:22.988263    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-495ebe-85c64f49b4-n9h8m"]
Oct 30 05:15:24.407296 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:24.407262    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35d1ddbc-3156-44bf-be1d-9f236418a2b7" path="/var/lib/kubelet/pods/35d1ddbc-3156-44bf-be1d-9f236418a2b7/volumes"
Oct 30 05:15:24.407691 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:24.407622    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c260ccd2-e290-4e56-84f0-2ebc79a5fcf7" path="/var/lib/kubelet/pods/c260ccd2-e290-4e56-84f0-2ebc79a5fcf7/volumes"
Oct 30 05:15:25.029554 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.029521    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"]
Oct 30 05:15:25.041956 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.041920    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f00c59-w9phh"]
Oct 30 05:15:25.279090 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.279022    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl" podUID="7ff7e01a-1fb4-44e9-b78d-0f59b26c4657" containerName="main" containerID="cri-o://4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80" gracePeriod=45
Oct 30 05:15:25.530370 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.530345    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 05:15:25.647605 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.647521    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-kube-api-access-wgrfl\" (UniqueName: \"kubernetes.io/projected/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-kube-api-access-wgrfl\") pod \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") "
Oct 30 05:15:25.648423 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.648399    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tmp-dir\") pod \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") "
Oct 30 05:15:25.648950 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.648460    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-home\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-home\") pod \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") "
Oct 30 05:15:25.648950 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.648520    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-model-cache\") pod \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") "
Oct 30 05:15:25.648950 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.648569    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-dshm\") pod \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") "
Oct 30 05:15:25.648950 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.648599    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tls-certs\") pod \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\" (UID: \"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657\") "
Oct 30 05:15:25.649369 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.649330    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-model-cache" pod "7ff7e01a-1fb4-44e9-b78d-0f59b26c4657" (UID: "7ff7e01a-1fb4-44e9-b78d-0f59b26c4657"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:25.649473 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.649403    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-home" pod "7ff7e01a-1fb4-44e9-b78d-0f59b26c4657" (UID: "7ff7e01a-1fb4-44e9-b78d-0f59b26c4657"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:25.651987 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.649604    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tmp-dir" pod "7ff7e01a-1fb4-44e9-b78d-0f59b26c4657" (UID: "7ff7e01a-1fb4-44e9-b78d-0f59b26c4657"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:25.651987 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.651902    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-kube-api-access-wgrfl" pod "7ff7e01a-1fb4-44e9-b78d-0f59b26c4657" (UID: "7ff7e01a-1fb4-44e9-b78d-0f59b26c4657"). InnerVolumeSpecName "kube-api-access-wgrfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:25.651987 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.651944    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-dshm" pod "7ff7e01a-1fb4-44e9-b78d-0f59b26c4657" (UID: "7ff7e01a-1fb4-44e9-b78d-0f59b26c4657"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:25.654411 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.654371    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tls-certs" pod "7ff7e01a-1fb4-44e9-b78d-0f59b26c4657" (UID: "7ff7e01a-1fb4-44e9-b78d-0f59b26c4657"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:15:25.749895 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.749863    5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:25.749895 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.749889    5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:25.749895 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.749899    5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:25.750157 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.749908    5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:25.750157 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.749915    5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:25.750157 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.749924    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgrfl\" (UniqueName: \"kubernetes.io/projected/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657-kube-api-access-wgrfl\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:25.906498 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.906402    5072 generic.go:356] "Generic (PLEG): container finished" podID="7ff7e01a-1fb4-44e9-b78d-0f59b26c4657" containerID="4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80" exitCode=0
Oct 30 05:15:25.906498 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.906451    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"
Oct 30 05:15:25.906732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.906491    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl" event={"ID":"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657","Type":"ContainerDied","Data":"4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80"}
Oct 30 05:15:25.906732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.906535    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl" event={"ID":"7ff7e01a-1fb4-44e9-b78d-0f59b26c4657","Type":"ContainerDied","Data":"b1921ff84c10b7610f2972f1eaba07fc799e24d2e5b887eadec646dafbbd0d7d"}
Oct 30 05:15:25.906732 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.906553    5072 scope.go:122] "RemoveContainer" containerID="4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80"
Oct 30 05:15:25.916801 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.916782    5072 scope.go:122] "RemoveContainer" containerID="4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80"
Oct 30 05:15:25.917122 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:25.917101    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80\": container with ID starting with 4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80 not found: ID does not exist" containerID="4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80"
Oct 30 05:15:25.917176 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.917134    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80"} err="failed to get container status \"4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80\": rpc error: code = NotFound desc = could not find container \"4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80\": container with ID starting with 4663b0dcf7486dd52e8874672268688b853043fe0e2160a6dc6080ac8528bf80 not found: ID does not exist"
Oct 30 05:15:25.960557 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.960531    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"]
Oct 30 05:15:25.963540 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:25.963519    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/distinct-w3-b4708a-kserve-775fdcd64-f6szl"]
Oct 30 05:15:26.407811 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:26.407778    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff7e01a-1fb4-44e9-b78d-0f59b26c4657" path="/var/lib/kubelet/pods/7ff7e01a-1fb4-44e9-b78d-0f59b26c4657/volumes"
Oct 30 05:15:26.408260 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:26.408245    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87350580-849d-40cd-a8f6-618c56f16493" path="/var/lib/kubelet/pods/87350580-849d-40cd-a8f6-618c56f16493/volumes"
Oct 30 05:15:34.051670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:34.051630    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"]
Oct 30 05:15:34.052199 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:34.051890    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q" podUID="e5255247-1989-4d28-a995-f5256f07e2db" containerName="istio-proxy" containerID="cri-o://091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761" gracePeriod=30
Oct 30 05:15:34.960177 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:34.960143    5072 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:15:34.960362 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:34.960204    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q" podUID="e5255247-1989-4d28-a995-f5256f07e2db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:15:36.032810 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:36.032778    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"]
Oct 30 05:15:36.045902 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:36.045876    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45642b14-vhld2"]
Oct 30 05:15:36.407505 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:36.407429    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8824e440-c110-4c3d-a1bc-067d8032310a" path="/var/lib/kubelet/pods/8824e440-c110-4c3d-a1bc-067d8032310a/volumes"
Oct 30 05:15:36.460253 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:36.460226    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mjlc5_b85dd801-9b87-496b-b22a-ca557d3fd2c4/manager/0.log"
Oct 30 05:15:37.607378 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:37.607313    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6fxf_a80b6be8-ec4e-40c6-8889-256a744263b3/manager/0.log"
Oct 30 05:15:37.798563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:37.798531    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qgl9w_a3f48faf-a0f6-4a0a-ac98-963183990dab/manager/0.log"
Oct 30 05:15:38.017675 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:38.017640    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-k6qmc_6671eae0-dd1d-41f5-b479-2c8e36e14650/manager/0.log"
Oct 30 05:15:38.463305 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:38.463273    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f4559f84c-tknlp_60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7/manager/0.log"
Oct 30 05:15:38.657203 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:38.657175    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g9hm4_812448a2-1e32-4f19-86f0-8cf17f32edd1/server/0.log"
Oct 30 05:15:39.068193 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.067683    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-mb5bq_c7bec9d6-a8f4-4051-867a-b28db0efb1ed/proxy/0.log"
Oct 30 05:15:39.318458 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.318401    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 05:15:39.392346 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.392319    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-credential-socket\") pod \"e5255247-1989-4d28-a995-f5256f07e2db\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") "
Oct 30 05:15:39.392472 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.392365    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5255247-1989-4d28-a995-f5256f07e2db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5255247-1989-4d28-a995-f5256f07e2db-istio-podinfo\") pod \"e5255247-1989-4d28-a995-f5256f07e2db\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") "
Oct 30 05:15:39.392472 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.392404    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-certs\") pod \"e5255247-1989-4d28-a995-f5256f07e2db\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") "
Oct 30 05:15:39.392472 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.392450    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-kube-api-access-2vtcj\" (UniqueName: \"kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-kube-api-access-2vtcj\") pod \"e5255247-1989-4d28-a995-f5256f07e2db\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") "
Oct 30 05:15:39.392631 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.392590    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-envoy\") pod \"e5255247-1989-4d28-a995-f5256f07e2db\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") "
Oct 30 05:15:39.392690 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.392625    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-credential-socket" pod "e5255247-1989-4d28-a995-f5256f07e2db" (UID: "e5255247-1989-4d28-a995-f5256f07e2db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:39.392755 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.392691    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-data\") pod \"e5255247-1989-4d28-a995-f5256f07e2db\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") "
Oct 30 05:15:39.392755 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.392740    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5255247-1989-4d28-a995-f5256f07e2db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5255247-1989-4d28-a995-f5256f07e2db-istiod-ca-cert\") pod \"e5255247-1989-4d28-a995-f5256f07e2db\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") "
Oct 30 05:15:39.392863 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.392761    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-certs" pod "e5255247-1989-4d28-a995-f5256f07e2db" (UID: "e5255247-1989-4d28-a995-f5256f07e2db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:39.392863 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.392821    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-istio-token\" (UniqueName: \"kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-istio-token\") pod \"e5255247-1989-4d28-a995-f5256f07e2db\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") "
Oct 30 05:15:39.392966 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.392866    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-socket\") pod \"e5255247-1989-4d28-a995-f5256f07e2db\" (UID: \"e5255247-1989-4d28-a995-f5256f07e2db\") "
Oct 30 05:15:39.393018 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.392959    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-data" pod "e5255247-1989-4d28-a995-f5256f07e2db" (UID: "e5255247-1989-4d28-a995-f5256f07e2db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:39.393607 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.393285    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5255247-1989-4d28-a995-f5256f07e2db-istiod-ca-cert" pod "e5255247-1989-4d28-a995-f5256f07e2db" (UID: "e5255247-1989-4d28-a995-f5256f07e2db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:39.393607 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.393317    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:39.393607 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.393340    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:39.393607 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.393353    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:39.393607 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.393567    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-socket" pod "e5255247-1989-4d28-a995-f5256f07e2db" (UID: "e5255247-1989-4d28-a995-f5256f07e2db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:39.395181 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.395157    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-istio-token" pod "e5255247-1989-4d28-a995-f5256f07e2db" (UID: "e5255247-1989-4d28-a995-f5256f07e2db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:39.395425 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.395394    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5255247-1989-4d28-a995-f5256f07e2db-istio-podinfo" pod "e5255247-1989-4d28-a995-f5256f07e2db" (UID: "e5255247-1989-4d28-a995-f5256f07e2db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:15:39.395549 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.395430    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-kube-api-access-2vtcj" pod "e5255247-1989-4d28-a995-f5256f07e2db" (UID: "e5255247-1989-4d28-a995-f5256f07e2db"). InnerVolumeSpecName "kube-api-access-2vtcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:39.395780 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.395754    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-envoy" pod "e5255247-1989-4d28-a995-f5256f07e2db" (UID: "e5255247-1989-4d28-a995-f5256f07e2db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:39.494147 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.494121    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5255247-1989-4d28-a995-f5256f07e2db-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:39.494147 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.494143    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:39.494147 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.494153    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:39.494380 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.494162    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5255247-1989-4d28-a995-f5256f07e2db-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:39.494380 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.494171    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vtcj\" (UniqueName: \"kubernetes.io/projected/e5255247-1989-4d28-a995-f5256f07e2db-kube-api-access-2vtcj\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:39.494380 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:39.494179    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5255247-1989-4d28-a995-f5256f07e2db-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:40.002223 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:40.002189    5072 generic.go:356] "Generic (PLEG): container finished" podID="e5255247-1989-4d28-a995-f5256f07e2db" containerID="091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761" exitCode=0
Oct 30 05:15:40.002670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:40.002239    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"
Oct 30 05:15:40.002670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:40.002275    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q" event={"ID":"e5255247-1989-4d28-a995-f5256f07e2db","Type":"ContainerDied","Data":"091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761"}
Oct 30 05:15:40.002670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:40.002334    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q" event={"ID":"e5255247-1989-4d28-a995-f5256f07e2db","Type":"ContainerDied","Data":"99d8c074b3eab23e5ed3670e735d4f260c42170515ee033df692551cda469d10"}
Oct 30 05:15:40.002670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:40.002357    5072 scope.go:122] "RemoveContainer" containerID="091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761"
Oct 30 05:15:40.022969 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:40.022947    5072 scope.go:122] "RemoveContainer" containerID="091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761"
Oct 30 05:15:40.023309 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:40.023285    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761\": container with ID starting with 091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761 not found: ID does not exist" containerID="091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761"
Oct 30 05:15:40.023388 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:40.023319    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761"} err="failed to get container status \"091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761\": rpc error: code = NotFound desc = could not find container \"091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761\": container with ID starting with 091a7bb8087dc1477265e69433776fddbf7ae67a81732dd0724f09c5f9b0d761 not found: ID does not exist"
Oct 30 05:15:40.071466 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:40.071428    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"]
Oct 30 05:15:40.084705 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:40.084680    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-x6c5q"]
Oct 30 05:15:40.409421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:40.409349    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5255247-1989-4d28-a995-f5256f07e2db" path="/var/lib/kubelet/pods/e5255247-1989-4d28-a995-f5256f07e2db/volumes"
Oct 30 05:15:41.750630 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:41.750593    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:15:43.733221 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:43.733189    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"]
Oct 30 05:15:43.733690 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:43.733377    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" podUID="6a5a918c-313e-446c-be37-2f71e771ca98" containerName="istio-proxy" containerID="cri-o://03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f" gracePeriod=30
Oct 30 05:15:46.044969 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:46.044936    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"]
Oct 30 05:15:46.057406 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:46.057366    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89987aa0-k65b9"]
Oct 30 05:15:46.407305 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:46.407222    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c052089a-2b41-415e-ab0c-cacefb08d619" path="/var/lib/kubelet/pods/c052089a-2b41-415e-ab0c-cacefb08d619/volumes"
Oct 30 05:15:47.031967 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:47.031931    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"]
Oct 30 05:15:47.044583 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:47.044551    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d86dd9-bhqtv"]
Oct 30 05:15:48.407577 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:48.407543    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5602814-ae4e-4a87-8ed6-06f42807d690" path="/var/lib/kubelet/pods/f5602814-ae4e-4a87-8ed6-06f42807d690/volumes"
Oct 30 05:15:49.019925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.019901    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:15:49.060526 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.060498    5072 generic.go:356] "Generic (PLEG): container finished" podID="6a5a918c-313e-446c-be37-2f71e771ca98" containerID="03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f" exitCode=0
Oct 30 05:15:49.060655 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.060545    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"
Oct 30 05:15:49.060655 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.060593    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" event={"ID":"6a5a918c-313e-446c-be37-2f71e771ca98","Type":"ContainerDied","Data":"03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f"}
Oct 30 05:15:49.060655 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.060642    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc" event={"ID":"6a5a918c-313e-446c-be37-2f71e771ca98","Type":"ContainerDied","Data":"f7243fc77a156562ab277bcfa208edfbd8afb52ac9522c7c6bd3d7cd7fe902e0"}
Oct 30 05:15:49.060766 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.060670    5072 scope.go:122] "RemoveContainer" containerID="03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f"
Oct 30 05:15:49.070788 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.070768    5072 scope.go:122] "RemoveContainer" containerID="03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f"
Oct 30 05:15:49.071069 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:15:49.071050    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f\": container with ID starting with 03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f not found: ID does not exist" containerID="03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f"
Oct 30 05:15:49.071173 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.071097    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f"} err="failed to get container status \"03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f\": rpc error: code = NotFound desc = could not find container \"03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f\": container with ID starting with 03a0960cc3ea1dfbe6cf5af28eb1638867b6294c1184d3b1234ccad20b138e1f not found: ID does not exist"
Oct 30 05:15:49.194844 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.194811    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-kube-api-access-kvkzk\" (UniqueName: \"kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-kube-api-access-kvkzk\") pod \"6a5a918c-313e-446c-be37-2f71e771ca98\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") "
Oct 30 05:15:49.195021 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.194887    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-certs\") pod \"6a5a918c-313e-446c-be37-2f71e771ca98\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") "
Oct 30 05:15:49.195021 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.194954    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a5a918c-313e-446c-be37-2f71e771ca98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5a918c-313e-446c-be37-2f71e771ca98-istio-podinfo\") pod \"6a5a918c-313e-446c-be37-2f71e771ca98\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") "
Oct 30 05:15:49.195021 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.194988    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-socket\") pod \"6a5a918c-313e-446c-be37-2f71e771ca98\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") "
Oct 30 05:15:49.195021 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.195014    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-data\") pod \"6a5a918c-313e-446c-be37-2f71e771ca98\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") "
Oct 30 05:15:49.195282 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.195116    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-istio-token\" (UniqueName: \"kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-istio-token\") pod \"6a5a918c-313e-446c-be37-2f71e771ca98\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") "
Oct 30 05:15:49.195282 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.195163    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5a918c-313e-446c-be37-2f71e771ca98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a918c-313e-446c-be37-2f71e771ca98-istiod-ca-cert\") pod \"6a5a918c-313e-446c-be37-2f71e771ca98\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") "
Oct 30 05:15:49.195282 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.195252    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-credential-socket\") pod \"6a5a918c-313e-446c-be37-2f71e771ca98\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") "
Oct 30 05:15:49.195442 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.195289    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-envoy\") pod \"6a5a918c-313e-446c-be37-2f71e771ca98\" (UID: \"6a5a918c-313e-446c-be37-2f71e771ca98\") "
Oct 30 05:15:49.195442 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.195295    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-socket" pod "6a5a918c-313e-446c-be37-2f71e771ca98" (UID: "6a5a918c-313e-446c-be37-2f71e771ca98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:49.195442 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.195312    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-data" pod "6a5a918c-313e-446c-be37-2f71e771ca98" (UID: "6a5a918c-313e-446c-be37-2f71e771ca98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:49.195595 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.195520    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-credential-socket" pod "6a5a918c-313e-446c-be37-2f71e771ca98" (UID: "6a5a918c-313e-446c-be37-2f71e771ca98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:49.195899 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.195776    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5a918c-313e-446c-be37-2f71e771ca98-istiod-ca-cert" pod "6a5a918c-313e-446c-be37-2f71e771ca98" (UID: "6a5a918c-313e-446c-be37-2f71e771ca98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:49.196068 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.195971    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-certs" pod "6a5a918c-313e-446c-be37-2f71e771ca98" (UID: "6a5a918c-313e-446c-be37-2f71e771ca98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:49.196182 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.196113    5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-credential-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:49.196182 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.196133    5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-socket\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:49.196182 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.196147    5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-data\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:49.197802 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.197782    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-kube-api-access-kvkzk" pod "6a5a918c-313e-446c-be37-2f71e771ca98" (UID: "6a5a918c-313e-446c-be37-2f71e771ca98"). InnerVolumeSpecName "kube-api-access-kvkzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:49.197935 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.197921    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a5a918c-313e-446c-be37-2f71e771ca98-istio-podinfo" pod "6a5a918c-313e-446c-be37-2f71e771ca98" (UID: "6a5a918c-313e-446c-be37-2f71e771ca98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:15:49.198098 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.198059    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-envoy" pod "6a5a918c-313e-446c-be37-2f71e771ca98" (UID: "6a5a918c-313e-446c-be37-2f71e771ca98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:49.198172 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.198120    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-istio-token" pod "6a5a918c-313e-446c-be37-2f71e771ca98" (UID: "6a5a918c-313e-446c-be37-2f71e771ca98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:49.296821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.296760    5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-workload-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:49.296821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.296787    5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5a918c-313e-446c-be37-2f71e771ca98-istio-podinfo\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:49.296821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.296800    5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-istio-token\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:49.296821 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.296811    5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a918c-313e-446c-be37-2f71e771ca98-istiod-ca-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:49.297019 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.296824    5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5a918c-313e-446c-be37-2f71e771ca98-istio-envoy\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:49.297019 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.296837    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvkzk\" (UniqueName: \"kubernetes.io/projected/6a5a918c-313e-446c-be37-2f71e771ca98-kube-api-access-kvkzk\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:15:49.411559 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.411516    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"]
Oct 30 05:15:49.424493 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:49.424463    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pghfc"]
Oct 30 05:15:50.408474 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:50.408439    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5a918c-313e-446c-be37-2f71e771ca98" path="/var/lib/kubelet/pods/6a5a918c-313e-446c-be37-2f71e771ca98/volumes"
Oct 30 05:15:57.027933 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:57.027888    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"]
Oct 30 05:15:57.042150 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:57.042114    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2187543e-5mtqs"]
Oct 30 05:15:58.407362 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:15:58.407327    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76256e9-dc69-471a-8542-59c49ccc5127" path="/var/lib/kubelet/pods/d76256e9-dc69-471a-8542-59c49ccc5127/volumes"
Oct 30 05:16:07.027444 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:07.027412    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"]
Oct 30 05:16:07.040001 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:07.039974    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f59bf4-847q2"]
Oct 30 05:16:08.407511 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:08.407479    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="017e8d82-8226-48a8-b622-d7117c705e16" path="/var/lib/kubelet/pods/017e8d82-8226-48a8-b622-d7117c705e16/volumes"
Oct 30 05:16:11.751224 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:16:11.751191    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:16:21.028780 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:21.028745    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"]
Oct 30 05:16:21.041088 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:21.041058    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a202e63-k2nf5"]
Oct 30 05:16:22.407827 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:22.407787    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34cbb344-363e-4b10-91cf-660d41431ccd" path="/var/lib/kubelet/pods/34cbb344-363e-4b10-91cf-660d41431ccd/volumes"
Oct 30 05:16:23.041977 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.041942    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f4559f84c-tknlp"]
Oct 30 05:16:23.042247 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.042200    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6f4559f84c-tknlp" podUID="60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7" containerName="manager" containerID="cri-o://cdc0e665c1047599ed6231045079daac0dd7daa02b4cc066aff27b1f9f4cdf51" gracePeriod=30
Oct 30 05:16:23.279453 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.279418    5072 generic.go:356] "Generic (PLEG): container finished" podID="60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7" containerID="cdc0e665c1047599ed6231045079daac0dd7daa02b4cc066aff27b1f9f4cdf51" exitCode=0
Oct 30 05:16:23.279616 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.279454    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4559f84c-tknlp" event={"ID":"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7","Type":"ContainerDied","Data":"cdc0e665c1047599ed6231045079daac0dd7daa02b4cc066aff27b1f9f4cdf51"}
Oct 30 05:16:23.315928 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.315904    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 05:16:23.443956 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.443928    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-webhook-cert\" (UniqueName: \"kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-webhook-cert\") pod \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\" (UID: \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\") "
Oct 30 05:16:23.444375 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.443986    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-metrics-certs\") pod \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\" (UID: \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\") "
Oct 30 05:16:23.444375 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.444023    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-kube-api-access-wgdkw\" (UniqueName: \"kubernetes.io/projected/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-kube-api-access-wgdkw\") pod \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\" (UID: \"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7\") "
Oct 30 05:16:23.446280 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.446251    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-kube-api-access-wgdkw" pod "60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7" (UID: "60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7"). InnerVolumeSpecName "kube-api-access-wgdkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:16:23.446280 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.446250    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-metrics-certs" pod "60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7" (UID: "60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:16:23.446280 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.446268    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-webhook-cert" pod "60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7" (UID: "60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:16:23.545363 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.545277    5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-metrics-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:16:23.545363 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.545310    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgdkw\" (UniqueName: \"kubernetes.io/projected/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-kube-api-access-wgdkw\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:16:23.545363 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:23.545322    5072 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7-webhook-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:16:24.288584 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:24.288545    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4559f84c-tknlp"
Oct 30 05:16:24.288584 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:24.288555    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4559f84c-tknlp" event={"ID":"60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7","Type":"ContainerDied","Data":"46071c9cbdb1bdd93248a1453374fee2efb481c18c1b19f3b733d0fac4067d7b"}
Oct 30 05:16:24.288867 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:24.288632    5072 scope.go:122] "RemoveContainer" containerID="cdc0e665c1047599ed6231045079daac0dd7daa02b4cc066aff27b1f9f4cdf51"
Oct 30 05:16:24.343995 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:24.343962    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f4559f84c-tknlp"]
Oct 30 05:16:24.355161 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:24.355140    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f4559f84c-tknlp"]
Oct 30 05:16:24.413381 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:24.413347    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7" path="/var/lib/kubelet/pods/60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7/volumes"
Oct 30 05:16:27.027619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:27.027584    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"]
Oct 30 05:16:27.039972 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:27.039946    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e311ac33-n99fv"]
Oct 30 05:16:28.407712 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:28.407680    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2446496a-8472-4ded-b6f5-b4ef56f518b7" path="/var/lib/kubelet/pods/2446496a-8472-4ded-b6f5-b4ef56f518b7/volumes"
Oct 30 05:16:31.029483 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:31.029455    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"]
Oct 30 05:16:31.041616 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:31.041580    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9f72815-9sd59"]
Oct 30 05:16:32.408066 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:32.408035    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e332a2-673b-4b12-94e7-48aec1d51716" path="/var/lib/kubelet/pods/f8e332a2-673b-4b12-94e7-48aec1d51716/volumes"
Oct 30 05:16:41.751436 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:16:41.751389    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:16:58.976249 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976203    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"]
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976752    5072 cpu_manager.go:414] "containerMap: removing container" podUID="dc6cf57f-0ddb-4e76-a06b-1cddd5e24070" containerName="main"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976764    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6cf57f-0ddb-4e76-a06b-1cddd5e24070" containerName="main"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976776    5072 cpu_manager.go:414] "containerMap: removing container" podUID="628d4322-92fa-4b80-9314-b2c8ca202561" containerName="cleanup"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976782    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628d4322-92fa-4b80-9314-b2c8ca202561" containerName="cleanup"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976800    5072 cpu_manager.go:414] "containerMap: removing container" podUID="ed107f59-f9a9-4a99-8a89-26a288611dd1" containerName="payload-pre-processing"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976806    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed107f59-f9a9-4a99-8a89-26a288611dd1" containerName="payload-pre-processing"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976819    5072 cpu_manager.go:414] "containerMap: removing container" podUID="b2bf0a67-0a7e-46cf-a27c-2789223e072e" containerName="revoke-keys"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976824    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bf0a67-0a7e-46cf-a27c-2789223e072e" containerName="revoke-keys"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976836    5072 cpu_manager.go:414] "containerMap: removing container" podUID="c260ccd2-e290-4e56-84f0-2ebc79a5fcf7" containerName="payload-processing"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976840    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c260ccd2-e290-4e56-84f0-2ebc79a5fcf7" containerName="payload-processing"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976852    5072 cpu_manager.go:414] "containerMap: removing container" podUID="6a5a918c-313e-446c-be37-2f71e771ca98" containerName="istio-proxy"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976856    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5a918c-313e-446c-be37-2f71e771ca98" containerName="istio-proxy"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976869    5072 cpu_manager.go:414] "containerMap: removing container" podUID="e5255247-1989-4d28-a995-f5256f07e2db" containerName="istio-proxy"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976875    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5255247-1989-4d28-a995-f5256f07e2db" containerName="istio-proxy"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976882    5072 cpu_manager.go:414] "containerMap: removing container" podUID="7ff7e01a-1fb4-44e9-b78d-0f59b26c4657" containerName="main"
Oct 30 05:16:58.976882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976887    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff7e01a-1fb4-44e9-b78d-0f59b26c4657" containerName="main"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976902    5072 cpu_manager.go:414] "containerMap: removing container" podUID="60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7" containerName="manager"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976906    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7" containerName="manager"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976917    5072 cpu_manager.go:414] "containerMap: removing container" podUID="628d4322-92fa-4b80-9314-b2c8ca202561" containerName="cleanup"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976924    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628d4322-92fa-4b80-9314-b2c8ca202561" containerName="cleanup"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976933    5072 cpu_manager.go:414] "containerMap: removing container" podUID="4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" containerName="istio-proxy"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976940    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" containerName="istio-proxy"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976952    5072 cpu_manager.go:414] "containerMap: removing container" podUID="35d1ddbc-3156-44bf-be1d-9f236418a2b7" containerName="payload-pre-processing"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.976959    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35d1ddbc-3156-44bf-be1d-9f236418a2b7" containerName="payload-pre-processing"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977048    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5a918c-313e-446c-be37-2f71e771ca98" containerName="istio-proxy"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977059    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5255247-1989-4d28-a995-f5256f07e2db" containerName="istio-proxy"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977065    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff7e01a-1fb4-44e9-b78d-0f59b26c4657" containerName="main"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977089    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ebfdc6-a13a-45ed-9ed8-e8cbaface1d7" containerName="manager"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977101    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="628d4322-92fa-4b80-9314-b2c8ca202561" containerName="cleanup"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977110    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="628d4322-92fa-4b80-9314-b2c8ca202561" containerName="cleanup"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977119    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dbdae54-cfe9-4ced-a212-2cfc4e1372ce" containerName="istio-proxy"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977127    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="35d1ddbc-3156-44bf-be1d-9f236418a2b7" containerName="payload-pre-processing"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977133    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6cf57f-0ddb-4e76-a06b-1cddd5e24070" containerName="main"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977139    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="628d4322-92fa-4b80-9314-b2c8ca202561" containerName="cleanup"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977145    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed107f59-f9a9-4a99-8a89-26a288611dd1" containerName="payload-pre-processing"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977152    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bf0a67-0a7e-46cf-a27c-2789223e072e" containerName="revoke-keys"
Oct 30 05:16:58.977764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.977161    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c260ccd2-e290-4e56-84f0-2ebc79a5fcf7" containerName="payload-processing"
Oct 30 05:16:58.980850 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:58.980824    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.015457 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.015421    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"]
Oct 30 05:16:59.080536 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.080505    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.080694 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.080545    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.080694 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.080649    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.080694 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.080673    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac611251-3890-4b60-b3a3-fb444a6befbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.080806 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.080779    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j45cn\" (UniqueName: \"kubernetes.io/projected/ac611251-3890-4b60-b3a3-fb444a6befbc-kube-api-access-j45cn\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.080872 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.080860    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.080955 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.080940    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.080996 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.080968    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac611251-3890-4b60-b3a3-fb444a6befbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.080996 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.080987    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac611251-3890-4b60-b3a3-fb444a6befbc-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.182272 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182240    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.182469 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182280    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.182469 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182314    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac611251-3890-4b60-b3a3-fb444a6befbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.182469 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182333    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac611251-3890-4b60-b3a3-fb444a6befbc-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.182469 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182373    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.182469 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182400    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.182469 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182453    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.182818 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182495    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac611251-3890-4b60-b3a3-fb444a6befbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.182818 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182599    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j45cn\" (UniqueName: \"kubernetes.io/projected/ac611251-3890-4b60-b3a3-fb444a6befbc-kube-api-access-j45cn\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.182818 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182684    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.182818 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182792    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.182977 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182890    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.183023 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.182999    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.183317 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.183297    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac611251-3890-4b60-b3a3-fb444a6befbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.184888 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.184862    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac611251-3890-4b60-b3a3-fb444a6befbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.185026 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.185010    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac611251-3890-4b60-b3a3-fb444a6befbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.205776 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.205754    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac611251-3890-4b60-b3a3-fb444a6befbc-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.206070 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.206047    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j45cn\" (UniqueName: \"kubernetes.io/projected/ac611251-3890-4b60-b3a3-fb444a6befbc-kube-api-access-j45cn\") pod \"maas-default-gateway-openshift-default-7f694df8bc-smfn4\" (UID: \"ac611251-3890-4b60-b3a3-fb444a6befbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.296060 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.295989    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:16:59.615098 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.615052    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"]
Oct 30 05:16:59.615551 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:16:59.615531    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac611251_3890_4b60_b3a3_fb444a6befbc.slice/crio-3f601a1de2ab85ba5e1cbd5e9e797f5b0b5bb4854ca76c36c2042ac4ee4227ea WatchSource:0}: Error finding container 3f601a1de2ab85ba5e1cbd5e9e797f5b0b5bb4854ca76c36c2042ac4ee4227ea: Status 404 returned error can't find the container with id 3f601a1de2ab85ba5e1cbd5e9e797f5b0b5bb4854ca76c36c2042ac4ee4227ea
Oct 30 05:16:59.617799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.617759    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:16:59.617929 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.617849    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:16:59.617929 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:16:59.617891    5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 05:17:00.506012 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:00.505971    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4" event={"ID":"ac611251-3890-4b60-b3a3-fb444a6befbc","Type":"ContainerStarted","Data":"82a2c1dcbe0acf3a835d194638bbafe448d7c320cac1e79628fbade6fbde3530"}
Oct 30 05:17:00.506558 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:00.506036    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4" event={"ID":"ac611251-3890-4b60-b3a3-fb444a6befbc","Type":"ContainerStarted","Data":"3f601a1de2ab85ba5e1cbd5e9e797f5b0b5bb4854ca76c36c2042ac4ee4227ea"}
Oct 30 05:17:00.572458 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:00.572381    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4" podStartSLOduration=2.572362565 podStartE2EDuration="2.572362565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:17:00.569605124 +0000 UTC m=+4800.311028235" watchObservedRunningTime="2026-10-30 05:17:00.572362565 +0000 UTC m=+4800.313785664"
Oct 30 05:17:01.189817 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:01.189785    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:17:01.211537 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:01.211508    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:17:01.297130 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:01.297064    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:17:01.298817 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:01.298792    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-smfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Oct 30 05:17:01.298882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:01.298844    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4" podUID="ac611251-3890-4b60-b3a3-fb444a6befbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Oct 30 05:17:02.296389 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:02.296355    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-smfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Oct 30 05:17:02.296814 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:02.296415    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4" podUID="ac611251-3890-4b60-b3a3-fb444a6befbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Oct 30 05:17:03.296819 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:03.296785    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-smfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Oct 30 05:17:03.297252 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:03.296837    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4" podUID="ac611251-3890-4b60-b3a3-fb444a6befbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Oct 30 05:17:03.606391 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:03.606311    5072 scope.go:122] "RemoveContainer" containerID="1f2ab2daa4d56ee06d253ea3b5f0d7b67d8f4700e29a7d9accde3f33c9fe39b3"
Oct 30 05:17:03.616302 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:03.616283    5072 scope.go:122] "RemoveContainer" containerID="f9875cc1bab19b3fe3878871d90cb040239aaa9028658ed4efbafaf5a781ab81"
Oct 30 05:17:03.625100 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:03.625064    5072 scope.go:122] "RemoveContainer" containerID="f3dfb053bacd6436fcb51e072502fb797747fce6a8849fa9d4726054aade09c5"
Oct 30 05:17:03.633387 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:03.633366    5072 scope.go:122] "RemoveContainer" containerID="2296339ba877cfcd2edfc093bc78dfd58c0854d010d677807dc5f1f33f30ece9"
Oct 30 05:17:03.641621 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:03.641604    5072 scope.go:122] "RemoveContainer" containerID="d1c7c537e2e1b1c9b113e5b5f61299c183a1ee7d33b9b5ebe9a5fe34c83d8933"
Oct 30 05:17:03.649739 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:03.649721    5072 scope.go:122] "RemoveContainer" containerID="a05fe32b547d512fc3340b34a5757c28f9685a1b768be9ad084d81e5acb85709"
Oct 30 05:17:04.296963 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:04.296931    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-smfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Oct 30 05:17:04.297399 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:04.296988    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4" podUID="ac611251-3890-4b60-b3a3-fb444a6befbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Oct 30 05:17:05.296654 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:05.296612    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-smfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Oct 30 05:17:05.296831 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:05.296666    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4" podUID="ac611251-3890-4b60-b3a3-fb444a6befbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Oct 30 05:17:06.297228 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:06.297193    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-smfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Oct 30 05:17:06.297617 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:06.297248    5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4" podUID="ac611251-3890-4b60-b3a3-fb444a6befbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Oct 30 05:17:07.763870 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:07.763835    5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:17:07.764427 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:07.764404    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:17:07.765460 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:07.765437    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4"
Oct 30 05:17:11.751342 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:17:11.751259    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:17:41.752859 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:17:41.752819    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:17:47.037984 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:47.037952    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"]
Oct 30 05:17:47.050247 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:47.050218    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97624e5-tt946"]
Oct 30 05:17:48.406994 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:48.406959    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc24ba4-cf9c-465d-b768-e4b5ff10e90d" path="/var/lib/kubelet/pods/1dc24ba4-cf9c-465d-b768-e4b5ff10e90d/volumes"
Oct 30 05:17:57.029938 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:57.029901    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"]
Oct 30 05:17:57.042439 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:57.042407    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b74399-s2h4n"]
Oct 30 05:17:58.406823 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:17:58.406788    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db06dc72-0f0f-4630-8372-75ef73295abd" path="/var/lib/kubelet/pods/db06dc72-0f0f-4630-8372-75ef73295abd/volumes"
Oct 30 05:18:03.722087 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:03.722055    5072 scope.go:122] "RemoveContainer" containerID="0e9c9bb13179ebc9703849e7819952774dc4ec503cd221c760325831a031638d"
Oct 30 05:18:03.731521 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:03.731499    5072 scope.go:122] "RemoveContainer" containerID="423bac4f817e00192174a32b31dadab71d29bfb962f96cf4c8f22dc0607d68a3"
Oct 30 05:18:03.739967 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:03.739935    5072 scope.go:122] "RemoveContainer" containerID="42142b98d62d07177bb41c26d5081ab3e1c5bf627f17d5dd011f039d74bab3ae"
Oct 30 05:18:03.748052 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:03.748034    5072 scope.go:122] "RemoveContainer" containerID="5cbe8b370bf24c87f6b777b8a24b9d535a97fb658c6a2cef4719f6bd93fd93ec"
Oct 30 05:18:08.030834 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:08.030793    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"]
Oct 30 05:18:08.042198 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:08.042165    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17974c41-62nw8"]
Oct 30 05:18:08.407158 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:08.407060    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf04038-6fbc-4804-a889-5e98468e76f6" path="/var/lib/kubelet/pods/ccf04038-6fbc-4804-a889-5e98468e76f6/volumes"
Oct 30 05:18:11.751223 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:18:11.751183    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:18:19.046223 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:19.046191    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"]
Oct 30 05:18:19.058925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:19.058900    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e27989b-d5crc"]
Oct 30 05:18:20.407328 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:20.407294    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42bc572-4da4-4491-a459-c6a00e1fb298" path="/var/lib/kubelet/pods/e42bc572-4da4-4491-a459-c6a00e1fb298/volumes"
Oct 30 05:18:35.032314 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:35.032272    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"]
Oct 30 05:18:35.044878 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:35.044843    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65592f9a-g9jr5"]
Oct 30 05:18:36.410925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:36.410885    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf339b3-44ed-48da-93a0-da34264415e5" path="/var/lib/kubelet/pods/2cf339b3-44ed-48da-93a0-da34264415e5/volumes"
Oct 30 05:18:41.750920 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:18:41.750830    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:18:44.027494 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:44.027453    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"]
Oct 30 05:18:44.040245 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:44.040211    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a59d3779-qnt58"]
Oct 30 05:18:44.407693 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:44.407615    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23397a6f-a7de-442a-bf57-84599654632c" path="/var/lib/kubelet/pods/23397a6f-a7de-442a-bf57-84599654632c/volumes"
Oct 30 05:18:45.040016 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:45.039982    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"]
Oct 30 05:18:45.043724 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:45.043688    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cbbf9c-khfw4"]
Oct 30 05:18:46.415724 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:46.415691    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e030506b-4676-4e1f-831d-f5f9a1209ecd" path="/var/lib/kubelet/pods/e030506b-4676-4e1f-831d-f5f9a1209ecd/volumes"
Oct 30 05:18:55.028029 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:55.027961    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"]
Oct 30 05:18:55.030670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:55.030643    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a423c4-r5dzc"]
Oct 30 05:18:56.407181 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:18:56.407138    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7366ae8d-b60d-42bf-942f-16f3c2356fef" path="/var/lib/kubelet/pods/7366ae8d-b60d-42bf-942f-16f3c2356fef/volumes"
Oct 30 05:19:03.784026 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:19:03.783990    5072 scope.go:122] "RemoveContainer" containerID="861596cb8a0c59bbf2e5049af3dcc12ae9be0d8d2f452d21d65dba6471a7cf95"
Oct 30 05:19:03.793504 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:19:03.793481    5072 scope.go:122] "RemoveContainer" containerID="b1c18329c8cff6a5f6ae94ed0d115fcb253fa01ea5e3b006c0677f1b43800503"
Oct 30 05:19:11.751349 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:19:11.751309    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:19:41.751888 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:19:41.751852    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:20:03.821025 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:20:03.820991    5072 scope.go:122] "RemoveContainer" containerID="152b5146621cb162c60cf3bf74621575c8bf4a55f9c4538ba64a3068d1ceb205"
Oct 30 05:20:03.833194 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:20:03.833170    5072 scope.go:122] "RemoveContainer" containerID="55316dc830cd1569d31ff2cf7dc23c217e0f3afb923a10f7540288d1562fe091"
Oct 30 05:20:03.842248 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:20:03.842224    5072 scope.go:122] "RemoveContainer" containerID="b3dbcae6f54cb62d7c77291548e5a6dbe3861447a7fdbdde49af5f61db638d07"
Oct 30 05:20:03.851614 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:20:03.851590    5072 scope.go:122] "RemoveContainer" containerID="9bacafa18b102c126f71c499fc4bfc27643ed51dd7d367454d76ec3139971791"
Oct 30 05:20:03.860180 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:20:03.860149    5072 scope.go:122] "RemoveContainer" containerID="52576ab0408f0ad81c0434d9d49fa22546178e0d5b515e407aedec68feaaf581"
Oct 30 05:20:03.868718 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:20:03.868695    5072 scope.go:122] "RemoveContainer" containerID="98dc15f55254e671a509f9a03773ce740bc8dc2e801b301511f32437d8ce84c8"
Oct 30 05:20:05.046469 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:20:05.046428    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"]
Oct 30 05:20:05.059275 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:20:05.059242    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b68edfb1-vqnrk"]
Oct 30 05:20:06.407494 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:20:06.407452    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2bf0a67-0a7e-46cf-a27c-2789223e072e" path="/var/lib/kubelet/pods/b2bf0a67-0a7e-46cf-a27c-2789223e072e/volumes"
Oct 30 05:20:11.752781 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:20:11.752701    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:20:41.750394 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:20:41.750353    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:21:03.909634 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:21:03.909593    5072 scope.go:122] "RemoveContainer" containerID="8be449d695e36aa15f1c5afd4e708e51a54d32f78922781d4200595e005f1b0b"
Oct 30 05:21:11.756431 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:21:11.756396    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:21:41.750004 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:21:41.749919    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:22:01.227713 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:22:01.227685    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:22:01.248814 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:22:01.248789    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:22:11.765875 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:22:11.765829    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:22:41.751755 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:22:41.751712    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:23:11.751663 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:23:11.751584    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:23:41.757466 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:23:41.757421    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:24:11.750479 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:24:11.750441    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:24:41.750726 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:24:41.750645    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:25:11.750932 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:25:11.750896    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:25:41.751646 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:25:41.751608    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:26:11.750597 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:26:11.750516    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:26:20.382281 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:26:20.382237    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gl57g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:26:20.382812 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:26:20.382307    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" podUID="8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:26:41.754662 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:26:41.754609    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:27:01.275500 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:27:01.275456    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:27:01.287127 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:27:01.287100    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:27:11.751451 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:27:11.751411    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:27:35.390101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:27:35.383172    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gl57g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:27:35.390101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:27:35.383241    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" podUID="8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:27:41.750494 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:27:41.750409    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:28:11.763607 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:28:11.763560    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:28:41.755477 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:28:41.755434    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:29:11.750209 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:29:11.750132    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:29:41.750954 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:29:41.750915    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:30:11.751745 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:30:11.751706    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:30:41.751843 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:30:41.751752    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:31:11.751244 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:31:11.751209    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:31:41.751493 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:31:41.751456    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:32:01.312841 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:32:01.312806    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:32:01.325091 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:32:01.325061    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:32:11.752201 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:32:11.752113    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:32:41.751385 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:32:41.751348    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:33:11.752276 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:33:11.752229    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:33:17.366889 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:17.366842    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"]
Oct 30 05:33:17.367547 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:17.367121    5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr" podUID="1e06d327-663e-402e-86be-c4e3296c37f3" containerName="maas-api" containerID="cri-o://3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf" gracePeriod=30
Oct 30 05:33:18.880173 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:18.878959    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 05:33:18.987534 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:18.987487    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-maas-api-tls\") pod \"1e06d327-663e-402e-86be-c4e3296c37f3\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") "
Oct 30 05:33:18.993389 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:18.993364    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e06d327-663e-402e-86be-c4e3296c37f3-kube-api-access-d8cl5\" (UniqueName: \"kubernetes.io/projected/1e06d327-663e-402e-86be-c4e3296c37f3-kube-api-access-d8cl5\") pod \"1e06d327-663e-402e-86be-c4e3296c37f3\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") "
Oct 30 05:33:19.013507 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.013484    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e06d327-663e-402e-86be-c4e3296c37f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e06d327-663e-402e-86be-c4e3296c37f3-openshift-service-ca\") pod \"1e06d327-663e-402e-86be-c4e3296c37f3\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") "
Oct 30 05:33:19.020381 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.020358    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-metrics-certs\") pod \"1e06d327-663e-402e-86be-c4e3296c37f3\" (UID: \"1e06d327-663e-402e-86be-c4e3296c37f3\") "
Oct 30 05:33:19.026423 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.020264    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e06d327-663e-402e-86be-c4e3296c37f3-openshift-service-ca" pod "1e06d327-663e-402e-86be-c4e3296c37f3" (UID: "1e06d327-663e-402e-86be-c4e3296c37f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:33:19.026623 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.026204    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e06d327-663e-402e-86be-c4e3296c37f3-kube-api-access-d8cl5" pod "1e06d327-663e-402e-86be-c4e3296c37f3" (UID: "1e06d327-663e-402e-86be-c4e3296c37f3"). InnerVolumeSpecName "kube-api-access-d8cl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:19.026895 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.026865    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8cl5\" (UniqueName: \"kubernetes.io/projected/1e06d327-663e-402e-86be-c4e3296c37f3-kube-api-access-d8cl5\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:19.027048 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.027010    5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e06d327-663e-402e-86be-c4e3296c37f3-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:19.032904 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.032881    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-maas-api-tls" pod "1e06d327-663e-402e-86be-c4e3296c37f3" (UID: "1e06d327-663e-402e-86be-c4e3296c37f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:33:19.035262 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.035226    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-metrics-certs" pod "1e06d327-663e-402e-86be-c4e3296c37f3" (UID: "1e06d327-663e-402e-86be-c4e3296c37f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:33:19.128239 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.128153    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:19.128239 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.128188    5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e06d327-663e-402e-86be-c4e3296c37f3-metrics-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:19.226376 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.226336    5072 generic.go:356] "Generic (PLEG): container finished" podID="1e06d327-663e-402e-86be-c4e3296c37f3" containerID="3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf" exitCode=0
Oct 30 05:33:19.226576 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.226396    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"
Oct 30 05:33:19.226576 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.226427    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr" event={"ID":"1e06d327-663e-402e-86be-c4e3296c37f3","Type":"ContainerDied","Data":"3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf"}
Oct 30 05:33:19.226576 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.226500    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr" event={"ID":"1e06d327-663e-402e-86be-c4e3296c37f3","Type":"ContainerDied","Data":"e9d4ebbdcc74a07f2e63b06a0e8f27453cb6de7046ea3ab8d1df72c2d1aafa4a"}
Oct 30 05:33:19.226576 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.226535    5072 scope.go:122] "RemoveContainer" containerID="3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf"
Oct 30 05:33:19.244867 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.244840    5072 scope.go:122] "RemoveContainer" containerID="3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf"
Oct 30 05:33:19.245788 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:33:19.245217    5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf\": container with ID starting with 3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf not found: ID does not exist" containerID="3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf"
Oct 30 05:33:19.245788 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.245257    5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf"} err="failed to get container status \"3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf\": rpc error: code = NotFound desc = could not find container \"3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf\": container with ID starting with 3b617aabb728ab2641717fbf2626d2c50caa39fbc9798b06aed4fd4eb0669ccf not found: ID does not exist"
Oct 30 05:33:19.281744 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.281716    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"]
Oct 30 05:33:19.295945 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:19.295915    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7f7c4f4-htctr"]
Oct 30 05:33:20.408976 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.408941    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e06d327-663e-402e-86be-c4e3296c37f3" path="/var/lib/kubelet/pods/1e06d327-663e-402e-86be-c4e3296c37f3/volumes"
Oct 30 05:33:20.414721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.414693    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"]
Oct 30 05:33:20.415209 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.415195    5072 cpu_manager.go:414] "containerMap: removing container" podUID="1e06d327-663e-402e-86be-c4e3296c37f3" containerName="maas-api"
Oct 30 05:33:20.415260 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.415211    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e06d327-663e-402e-86be-c4e3296c37f3" containerName="maas-api"
Oct 30 05:33:20.415260 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.415246    5072 cpu_manager.go:414] "containerMap: removing container" podUID="628d4322-92fa-4b80-9314-b2c8ca202561" containerName="cleanup"
Oct 30 05:33:20.415260 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.415252    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628d4322-92fa-4b80-9314-b2c8ca202561" containerName="cleanup"
Oct 30 05:33:20.415364 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.415334    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e06d327-663e-402e-86be-c4e3296c37f3" containerName="maas-api"
Oct 30 05:33:20.419750 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.419735    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:20.448350 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.448210    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:33:20.448350 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.448288    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:33:20.448350 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.448304    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:33:20.448350 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.448333    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:33:20.450407 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.450383    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"]
Oct 30 05:33:20.543156 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.543128    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ff93de4-pf8dj\" (UID: \"da4e97a8-c2e7-4922-9b04-42ef3182518b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:20.543297 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.543255    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ff93de4-pf8dj\" (UID: \"da4e97a8-c2e7-4922-9b04-42ef3182518b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:20.543349 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.543334    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp7gc\" (UniqueName: \"kubernetes.io/projected/da4e97a8-c2e7-4922-9b04-42ef3182518b-kube-api-access-mp7gc\") pod \"maas-api-revoke-sub-3ff93de4-pf8dj\" (UID: \"da4e97a8-c2e7-4922-9b04-42ef3182518b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:20.644899 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.644869    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp7gc\" (UniqueName: \"kubernetes.io/projected/da4e97a8-c2e7-4922-9b04-42ef3182518b-kube-api-access-mp7gc\") pod \"maas-api-revoke-sub-3ff93de4-pf8dj\" (UID: \"da4e97a8-c2e7-4922-9b04-42ef3182518b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:20.645030 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.644918    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ff93de4-pf8dj\" (UID: \"da4e97a8-c2e7-4922-9b04-42ef3182518b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:20.645030 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.645009    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ff93de4-pf8dj\" (UID: \"da4e97a8-c2e7-4922-9b04-42ef3182518b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:20.645687 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.645668    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ff93de4-pf8dj\" (UID: \"da4e97a8-c2e7-4922-9b04-42ef3182518b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:20.647430 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.647412    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ff93de4-pf8dj\" (UID: \"da4e97a8-c2e7-4922-9b04-42ef3182518b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:20.669497 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.669444    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp7gc\" (UniqueName: \"kubernetes.io/projected/da4e97a8-c2e7-4922-9b04-42ef3182518b-kube-api-access-mp7gc\") pod \"maas-api-revoke-sub-3ff93de4-pf8dj\" (UID: \"da4e97a8-c2e7-4922-9b04-42ef3182518b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:20.733233 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:20.733195    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:21.037577 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:21.037542    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"]
Oct 30 05:33:21.038369 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:33:21.038337    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4e97a8_c2e7_4922_9b04_42ef3182518b.slice/crio-b4efafd06e24f099650598828609219ca8602f9ff092ea6dc0ccfd149efa6cc3 WatchSource:0}: Error finding container b4efafd06e24f099650598828609219ca8602f9ff092ea6dc0ccfd149efa6cc3: Status 404 returned error can't find the container with id b4efafd06e24f099650598828609219ca8602f9ff092ea6dc0ccfd149efa6cc3
Oct 30 05:33:21.239596 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:21.239563    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj" event={"ID":"da4e97a8-c2e7-4922-9b04-42ef3182518b","Type":"ContainerStarted","Data":"b4efafd06e24f099650598828609219ca8602f9ff092ea6dc0ccfd149efa6cc3"}
Oct 30 05:33:22.245962 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:22.245930    5072 generic.go:356] "Generic (PLEG): container finished" podID="da4e97a8-c2e7-4922-9b04-42ef3182518b" containerID="0aeba166e60e857e7088b742d821c3cea8a4e6f821530533dbd8166e01de4655" exitCode=0
Oct 30 05:33:22.246383 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:22.246025    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj" event={"ID":"da4e97a8-c2e7-4922-9b04-42ef3182518b","Type":"ContainerDied","Data":"0aeba166e60e857e7088b742d821c3cea8a4e6f821530533dbd8166e01de4655"}
Oct 30 05:33:23.404520 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:23.404491    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:23.472373 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:23.472327    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4e97a8-c2e7-4922-9b04-42ef3182518b-kube-api-access-mp7gc\" (UniqueName: \"kubernetes.io/projected/da4e97a8-c2e7-4922-9b04-42ef3182518b-kube-api-access-mp7gc\") pod \"da4e97a8-c2e7-4922-9b04-42ef3182518b\" (UID: \"da4e97a8-c2e7-4922-9b04-42ef3182518b\") "
Oct 30 05:33:23.472581 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:23.472420    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-xks-ca\") pod \"da4e97a8-c2e7-4922-9b04-42ef3182518b\" (UID: \"da4e97a8-c2e7-4922-9b04-42ef3182518b\") "
Oct 30 05:33:23.472643 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:23.472624    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-openshift-ca\") pod \"da4e97a8-c2e7-4922-9b04-42ef3182518b\" (UID: \"da4e97a8-c2e7-4922-9b04-42ef3182518b\") "
Oct 30 05:33:23.473005 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:23.472971    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-openshift-ca" pod "da4e97a8-c2e7-4922-9b04-42ef3182518b" (UID: "da4e97a8-c2e7-4922-9b04-42ef3182518b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:33:23.473219 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:23.473191    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:23.474696 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:23.474667    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4e97a8-c2e7-4922-9b04-42ef3182518b-kube-api-access-mp7gc" pod "da4e97a8-c2e7-4922-9b04-42ef3182518b" (UID: "da4e97a8-c2e7-4922-9b04-42ef3182518b"). InnerVolumeSpecName "kube-api-access-mp7gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:23.475211 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:23.475190    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-xks-ca" pod "da4e97a8-c2e7-4922-9b04-42ef3182518b" (UID: "da4e97a8-c2e7-4922-9b04-42ef3182518b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:33:23.574640 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:23.574551    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp7gc\" (UniqueName: \"kubernetes.io/projected/da4e97a8-c2e7-4922-9b04-42ef3182518b-kube-api-access-mp7gc\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:23.574640 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:23.574587    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4e97a8-c2e7-4922-9b04-42ef3182518b-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:24.257264 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:24.257235    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"
Oct 30 05:33:24.257453 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:24.257275    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj" event={"ID":"da4e97a8-c2e7-4922-9b04-42ef3182518b","Type":"ContainerDied","Data":"b4efafd06e24f099650598828609219ca8602f9ff092ea6dc0ccfd149efa6cc3"}
Oct 30 05:33:24.257453 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:24.257330    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4efafd06e24f099650598828609219ca8602f9ff092ea6dc0ccfd149efa6cc3"
Oct 30 05:33:30.781385 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.781351    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"]
Oct 30 05:33:30.781905 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.781889    5072 cpu_manager.go:414] "containerMap: removing container" podUID="da4e97a8-c2e7-4922-9b04-42ef3182518b" containerName="revoke-keys"
Oct 30 05:33:30.781971 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.781907    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4e97a8-c2e7-4922-9b04-42ef3182518b" containerName="revoke-keys"
Oct 30 05:33:30.782010 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.781993    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4e97a8-c2e7-4922-9b04-42ef3182518b" containerName="revoke-keys"
Oct 30 05:33:30.785631 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.785610    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:30.806388 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.806358    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"]
Oct 30 05:33:30.816219 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.816192    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:33:30.816219 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.816197    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:33:30.816441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.816292    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:33:30.816441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.816410    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:33:30.843336 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.843297    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b66617-q5cb2\" (UID: \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:30.843495 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.843472    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnt2g\" (UniqueName: \"kubernetes.io/projected/2b6cc501-417e-4a72-9331-b8ea9354f2b2-kube-api-access-vnt2g\") pod \"maas-api-revoke-sub-85b66617-q5cb2\" (UID: \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:30.843567 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.843546    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b66617-q5cb2\" (UID: \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:30.944818 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.944783    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnt2g\" (UniqueName: \"kubernetes.io/projected/2b6cc501-417e-4a72-9331-b8ea9354f2b2-kube-api-access-vnt2g\") pod \"maas-api-revoke-sub-85b66617-q5cb2\" (UID: \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:30.945028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.944836    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b66617-q5cb2\" (UID: \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:30.945028 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.944880    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b66617-q5cb2\" (UID: \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:30.945552 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.945522    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b66617-q5cb2\" (UID: \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:30.947497 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.947474    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b66617-q5cb2\" (UID: \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:30.969986 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:30.969960    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnt2g\" (UniqueName: \"kubernetes.io/projected/2b6cc501-417e-4a72-9331-b8ea9354f2b2-kube-api-access-vnt2g\") pod \"maas-api-revoke-sub-85b66617-q5cb2\" (UID: \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:31.100044 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:31.099957    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:31.403486 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:31.403454    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"]
Oct 30 05:33:31.404966 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:33:31.404942    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6cc501_417e_4a72_9331_b8ea9354f2b2.slice/crio-020b6e5c381474d8091c5a4fde9315433b8bf9c16a9fc6b940edb61cae728770 WatchSource:0}: Error finding container 020b6e5c381474d8091c5a4fde9315433b8bf9c16a9fc6b940edb61cae728770: Status 404 returned error can't find the container with id 020b6e5c381474d8091c5a4fde9315433b8bf9c16a9fc6b940edb61cae728770
Oct 30 05:33:32.301866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:32.301825    5072 generic.go:356] "Generic (PLEG): container finished" podID="2b6cc501-417e-4a72-9331-b8ea9354f2b2" containerID="b393d31f96a5c4b2c0dee8b0c225cb350d787bfbc9477dd52bbfd9e311018bfe" exitCode=0
Oct 30 05:33:32.302314 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:32.301903    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2" event={"ID":"2b6cc501-417e-4a72-9331-b8ea9354f2b2","Type":"ContainerDied","Data":"b393d31f96a5c4b2c0dee8b0c225cb350d787bfbc9477dd52bbfd9e311018bfe"}
Oct 30 05:33:32.302314 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:32.301987    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2" event={"ID":"2b6cc501-417e-4a72-9331-b8ea9354f2b2","Type":"ContainerStarted","Data":"020b6e5c381474d8091c5a4fde9315433b8bf9c16a9fc6b940edb61cae728770"}
Oct 30 05:33:33.445203 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:33.445178    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:33.570399 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:33.570318    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-openshift-ca\") pod \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\" (UID: \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\") "
Oct 30 05:33:33.570399 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:33.570367    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6cc501-417e-4a72-9331-b8ea9354f2b2-kube-api-access-vnt2g\" (UniqueName: \"kubernetes.io/projected/2b6cc501-417e-4a72-9331-b8ea9354f2b2-kube-api-access-vnt2g\") pod \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\" (UID: \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\") "
Oct 30 05:33:33.570616 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:33.570470    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-xks-ca\") pod \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\" (UID: \"2b6cc501-417e-4a72-9331-b8ea9354f2b2\") "
Oct 30 05:33:33.570667 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:33.570645    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-openshift-ca" pod "2b6cc501-417e-4a72-9331-b8ea9354f2b2" (UID: "2b6cc501-417e-4a72-9331-b8ea9354f2b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:33:33.570824 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:33.570805    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:33.572776 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:33.572755    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6cc501-417e-4a72-9331-b8ea9354f2b2-kube-api-access-vnt2g" pod "2b6cc501-417e-4a72-9331-b8ea9354f2b2" (UID: "2b6cc501-417e-4a72-9331-b8ea9354f2b2"). InnerVolumeSpecName "kube-api-access-vnt2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:33.572864 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:33.572806    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-xks-ca" pod "2b6cc501-417e-4a72-9331-b8ea9354f2b2" (UID: "2b6cc501-417e-4a72-9331-b8ea9354f2b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:33:33.671643 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:33.671609    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6cc501-417e-4a72-9331-b8ea9354f2b2-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:33.671643 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:33.671637    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnt2g\" (UniqueName: \"kubernetes.io/projected/2b6cc501-417e-4a72-9331-b8ea9354f2b2-kube-api-access-vnt2g\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:34.313950 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:34.313920    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"
Oct 30 05:33:34.314150 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:34.313944    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2" event={"ID":"2b6cc501-417e-4a72-9331-b8ea9354f2b2","Type":"ContainerDied","Data":"020b6e5c381474d8091c5a4fde9315433b8bf9c16a9fc6b940edb61cae728770"}
Oct 30 05:33:34.314150 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:34.313991    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="020b6e5c381474d8091c5a4fde9315433b8bf9c16a9fc6b940edb61cae728770"
Oct 30 05:33:41.751412 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:33:41.751328    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:33:50.384102 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:50.383255    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gl57g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:33:50.384102 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:50.383347    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" podUID="8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:53.567206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.567168    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"]
Oct 30 05:33:53.567697 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.567674    5072 cpu_manager.go:414] "containerMap: removing container" podUID="2b6cc501-417e-4a72-9331-b8ea9354f2b2" containerName="revoke-keys"
Oct 30 05:33:53.567697 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.567690    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6cc501-417e-4a72-9331-b8ea9354f2b2" containerName="revoke-keys"
Oct 30 05:33:53.567882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.567781    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6cc501-417e-4a72-9331-b8ea9354f2b2" containerName="revoke-keys"
Oct 30 05:33:53.571039 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.571022    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:53.588302 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.588270    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:33:53.588533 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.588379    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:33:53.589067 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.589030    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:33:53.589220 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.589196    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:33:53.592392 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.592365    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"]
Oct 30 05:33:53.680274 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.680246    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-470c2cbb-5wznj\" (UID: \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:53.680470 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.680376    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-470c2cbb-5wznj\" (UID: \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:53.680470 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.680461    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6gfl\" (UniqueName: \"kubernetes.io/projected/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-kube-api-access-w6gfl\") pod \"maas-api-revoke-sub-470c2cbb-5wznj\" (UID: \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:53.781801 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.781770    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-470c2cbb-5wznj\" (UID: \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:53.781964 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.781845    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-470c2cbb-5wznj\" (UID: \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:53.781964 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.781880    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6gfl\" (UniqueName: \"kubernetes.io/projected/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-kube-api-access-w6gfl\") pod \"maas-api-revoke-sub-470c2cbb-5wznj\" (UID: \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:53.782604 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.782576    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-470c2cbb-5wznj\" (UID: \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:53.784341 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.784323    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-470c2cbb-5wznj\" (UID: \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:53.806528 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.806504    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6gfl\" (UniqueName: \"kubernetes.io/projected/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-kube-api-access-w6gfl\") pod \"maas-api-revoke-sub-470c2cbb-5wznj\" (UID: \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:53.889857 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:53.889773    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:54.192331 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:54.192300    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"]
Oct 30 05:33:54.192461 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:33:54.192319    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a0f085_bff2_442a_8af2_6bcf36d3f47f.slice/crio-bfd9f6150e06d2cbc7114c74947dbd2063885f1e0e979c512948a22612b7bb07 WatchSource:0}: Error finding container bfd9f6150e06d2cbc7114c74947dbd2063885f1e0e979c512948a22612b7bb07: Status 404 returned error can't find the container with id bfd9f6150e06d2cbc7114c74947dbd2063885f1e0e979c512948a22612b7bb07
Oct 30 05:33:54.458751 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:54.458713    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj" event={"ID":"c2a0f085-bff2-442a-8af2-6bcf36d3f47f","Type":"ContainerStarted","Data":"bfd9f6150e06d2cbc7114c74947dbd2063885f1e0e979c512948a22612b7bb07"}
Oct 30 05:33:55.466036 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:55.465999    5072 generic.go:356] "Generic (PLEG): container finished" podID="c2a0f085-bff2-442a-8af2-6bcf36d3f47f" containerID="590d59962ce0640fd242feabb277746c07fcbe84b1126bdfc2e5d4cffec92398" exitCode=0
Oct 30 05:33:55.466507 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:55.466109    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj" event={"ID":"c2a0f085-bff2-442a-8af2-6bcf36d3f47f","Type":"ContainerDied","Data":"590d59962ce0640fd242feabb277746c07fcbe84b1126bdfc2e5d4cffec92398"}
Oct 30 05:33:56.620975 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:56.620947    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:56.711210 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:56.711170    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-openshift-ca\") pod \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\" (UID: \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\") "
Oct 30 05:33:56.711392 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:56.711226    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-kube-api-access-w6gfl\" (UniqueName: \"kubernetes.io/projected/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-kube-api-access-w6gfl\") pod \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\" (UID: \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\") "
Oct 30 05:33:56.711392 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:56.711350    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-xks-ca\") pod \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\" (UID: \"c2a0f085-bff2-442a-8af2-6bcf36d3f47f\") "
Oct 30 05:33:56.711590 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:56.711568    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-openshift-ca" pod "c2a0f085-bff2-442a-8af2-6bcf36d3f47f" (UID: "c2a0f085-bff2-442a-8af2-6bcf36d3f47f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:33:56.711787 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:56.711763    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:56.713491 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:56.713463    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-xks-ca" pod "c2a0f085-bff2-442a-8af2-6bcf36d3f47f" (UID: "c2a0f085-bff2-442a-8af2-6bcf36d3f47f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:33:56.713575 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:56.713506    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-kube-api-access-w6gfl" pod "c2a0f085-bff2-442a-8af2-6bcf36d3f47f" (UID: "c2a0f085-bff2-442a-8af2-6bcf36d3f47f"). InnerVolumeSpecName "kube-api-access-w6gfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:56.812629 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:56.812564    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:56.812629 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:56.812587    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6gfl\" (UniqueName: \"kubernetes.io/projected/c2a0f085-bff2-442a-8af2-6bcf36d3f47f-kube-api-access-w6gfl\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:33:57.480668 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:57.480636    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"
Oct 30 05:33:57.480857 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:57.480665    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj" event={"ID":"c2a0f085-bff2-442a-8af2-6bcf36d3f47f","Type":"ContainerDied","Data":"bfd9f6150e06d2cbc7114c74947dbd2063885f1e0e979c512948a22612b7bb07"}
Oct 30 05:33:57.480857 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:33:57.480718    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfd9f6150e06d2cbc7114c74947dbd2063885f1e0e979c512948a22612b7bb07"
Oct 30 05:34:11.840502 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:34:11.840459    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:34:12.009105 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:12.005192    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-txlk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:34:12.009105 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:12.005277    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4" podUID="b748c4e2-2e95-43e6-8dc4-9bf3958a24d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:34:24.529342 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.529306    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"]
Oct 30 05:34:24.529837 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.529776    5072 cpu_manager.go:414] "containerMap: removing container" podUID="c2a0f085-bff2-442a-8af2-6bcf36d3f47f" containerName="revoke-keys"
Oct 30 05:34:24.529837 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.529787    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a0f085-bff2-442a-8af2-6bcf36d3f47f" containerName="revoke-keys"
Oct 30 05:34:24.529913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.529887    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a0f085-bff2-442a-8af2-6bcf36d3f47f" containerName="revoke-keys"
Oct 30 05:34:24.533235 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.533216    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:24.548481 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.548457    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:34:24.548654 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.548514    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:34:24.561348 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.561325    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:34:24.561348 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.561338    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:34:24.563870 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.563845    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"]
Oct 30 05:34:24.640568 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.640540    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef228b2-dvchg\" (UID: \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:24.640705 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.640602    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpd78\" (UniqueName: \"kubernetes.io/projected/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-kube-api-access-fpd78\") pod \"maas-api-revoke-sub-4ef228b2-dvchg\" (UID: \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:24.640782 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.640713    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef228b2-dvchg\" (UID: \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:24.742053 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.742018    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpd78\" (UniqueName: \"kubernetes.io/projected/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-kube-api-access-fpd78\") pod \"maas-api-revoke-sub-4ef228b2-dvchg\" (UID: \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:24.742270 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.742062    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef228b2-dvchg\" (UID: \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:24.742270 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.742200    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef228b2-dvchg\" (UID: \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:24.742834 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.742813    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef228b2-dvchg\" (UID: \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:24.744675 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.744652    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef228b2-dvchg\" (UID: \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:24.767726 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.767698    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpd78\" (UniqueName: \"kubernetes.io/projected/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-kube-api-access-fpd78\") pod \"maas-api-revoke-sub-4ef228b2-dvchg\" (UID: \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:24.847203 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:24.847144    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:25.148653 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:25.148625    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"]
Oct 30 05:34:25.149731 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:34:25.149699    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ce1164_ed6b_4751_98c6_b2e6641bf27d.slice/crio-dfa8172dae21dcb4fe1dea0991659cbf3329a37b8860dfe17096354e4f9a331a WatchSource:0}: Error finding container dfa8172dae21dcb4fe1dea0991659cbf3329a37b8860dfe17096354e4f9a331a: Status 404 returned error can't find the container with id dfa8172dae21dcb4fe1dea0991659cbf3329a37b8860dfe17096354e4f9a331a
Oct 30 05:34:25.675467 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:25.675426    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg" event={"ID":"f2ce1164-ed6b-4751-98c6-b2e6641bf27d","Type":"ContainerStarted","Data":"8f0b5f33fdec54985b30678765ec2b03374618ab1671b1b66b170ea3a4339545"}
Oct 30 05:34:25.675907 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:25.675483    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg" event={"ID":"f2ce1164-ed6b-4751-98c6-b2e6641bf27d","Type":"ContainerStarted","Data":"dfa8172dae21dcb4fe1dea0991659cbf3329a37b8860dfe17096354e4f9a331a"}
Oct 30 05:34:25.718511 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:25.718423    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg" podStartSLOduration=1.718406109 podStartE2EDuration="1.718406109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:34:25.717514143 +0000 UTC m=+5845.458937240" watchObservedRunningTime="2026-10-30 05:34:25.718406109 +0000 UTC m=+5845.459829210"
Oct 30 05:34:26.681736 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:26.681699    5072 generic.go:356] "Generic (PLEG): container finished" podID="f2ce1164-ed6b-4751-98c6-b2e6641bf27d" containerID="8f0b5f33fdec54985b30678765ec2b03374618ab1671b1b66b170ea3a4339545" exitCode=0
Oct 30 05:34:26.682215 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:26.681784    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg" event={"ID":"f2ce1164-ed6b-4751-98c6-b2e6641bf27d","Type":"ContainerDied","Data":"8f0b5f33fdec54985b30678765ec2b03374618ab1671b1b66b170ea3a4339545"}
Oct 30 05:34:27.824853 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:27.824826    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:27.976240 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:27.976162    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-kube-api-access-fpd78\" (UniqueName: \"kubernetes.io/projected/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-kube-api-access-fpd78\") pod \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\" (UID: \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\") "
Oct 30 05:34:27.976240 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:27.976218    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-xks-ca\") pod \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\" (UID: \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\") "
Oct 30 05:34:27.976463 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:27.976255    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-openshift-ca\") pod \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\" (UID: \"f2ce1164-ed6b-4751-98c6-b2e6641bf27d\") "
Oct 30 05:34:27.976689 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:27.976662    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-openshift-ca" pod "f2ce1164-ed6b-4751-98c6-b2e6641bf27d" (UID: "f2ce1164-ed6b-4751-98c6-b2e6641bf27d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:34:27.978549 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:27.978523    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-xks-ca" pod "f2ce1164-ed6b-4751-98c6-b2e6641bf27d" (UID: "f2ce1164-ed6b-4751-98c6-b2e6641bf27d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:34:27.978659 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:27.978578    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-kube-api-access-fpd78" pod "f2ce1164-ed6b-4751-98c6-b2e6641bf27d" (UID: "f2ce1164-ed6b-4751-98c6-b2e6641bf27d"). InnerVolumeSpecName "kube-api-access-fpd78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:34:28.078117 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:28.078061    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpd78\" (UniqueName: \"kubernetes.io/projected/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-kube-api-access-fpd78\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:34:28.078117 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:28.078114    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:34:28.078291 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:28.078126    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ce1164-ed6b-4751-98c6-b2e6641bf27d-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:34:28.693061 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:28.693034    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg"
Oct 30 05:34:28.693222 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:28.693030    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef228b2-dvchg" event={"ID":"f2ce1164-ed6b-4751-98c6-b2e6641bf27d","Type":"ContainerDied","Data":"dfa8172dae21dcb4fe1dea0991659cbf3329a37b8860dfe17096354e4f9a331a"}
Oct 30 05:34:28.693222 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:28.693138    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa8172dae21dcb4fe1dea0991659cbf3329a37b8860dfe17096354e4f9a331a"
Oct 30 05:34:34.909053 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:34.909018    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"]
Oct 30 05:34:34.909591 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:34.909575    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f2ce1164-ed6b-4751-98c6-b2e6641bf27d" containerName="revoke-keys"
Oct 30 05:34:34.909641 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:34.909593    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ce1164-ed6b-4751-98c6-b2e6641bf27d" containerName="revoke-keys"
Oct 30 05:34:34.909726 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:34.909716    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ce1164-ed6b-4751-98c6-b2e6641bf27d" containerName="revoke-keys"
Oct 30 05:34:34.913152 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:34.913130    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:34.941713 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:34.941685    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:34:34.941713 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:34.941684    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:34:34.941924 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:34.941743    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:34:34.941924 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:34.941758    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:34:34.944146 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:34.944121    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"]
Oct 30 05:34:35.050202 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:35.050169    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4ed2e9-wszd6\" (UID: \"115b8606-cbec-4642-9dc9-61abefa440af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:35.050377 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:35.050252    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4ed2e9-wszd6\" (UID: \"115b8606-cbec-4642-9dc9-61abefa440af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:35.050377 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:35.050335    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8t2j\" (UniqueName: \"kubernetes.io/projected/115b8606-cbec-4642-9dc9-61abefa440af-kube-api-access-z8t2j\") pod \"maas-api-revoke-sub-8d4ed2e9-wszd6\" (UID: \"115b8606-cbec-4642-9dc9-61abefa440af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:35.151603 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:35.151562    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4ed2e9-wszd6\" (UID: \"115b8606-cbec-4642-9dc9-61abefa440af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:35.151801 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:35.151665    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4ed2e9-wszd6\" (UID: \"115b8606-cbec-4642-9dc9-61abefa440af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:35.151801 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:35.151697    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8t2j\" (UniqueName: \"kubernetes.io/projected/115b8606-cbec-4642-9dc9-61abefa440af-kube-api-access-z8t2j\") pod \"maas-api-revoke-sub-8d4ed2e9-wszd6\" (UID: \"115b8606-cbec-4642-9dc9-61abefa440af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:35.152425 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:35.152335    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4ed2e9-wszd6\" (UID: \"115b8606-cbec-4642-9dc9-61abefa440af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:35.154215 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:35.154192    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4ed2e9-wszd6\" (UID: \"115b8606-cbec-4642-9dc9-61abefa440af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:35.176249 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:35.176186    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8t2j\" (UniqueName: \"kubernetes.io/projected/115b8606-cbec-4642-9dc9-61abefa440af-kube-api-access-z8t2j\") pod \"maas-api-revoke-sub-8d4ed2e9-wszd6\" (UID: \"115b8606-cbec-4642-9dc9-61abefa440af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:35.231862 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:35.231838    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:35.531991 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:35.531965    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"]
Oct 30 05:34:35.532383 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:34:35.532359    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115b8606_cbec_4642_9dc9_61abefa440af.slice/crio-231f5101024a82c58e876a4b91cb2af968971b93dff119c379a7825a85a77c88 WatchSource:0}: Error finding container 231f5101024a82c58e876a4b91cb2af968971b93dff119c379a7825a85a77c88: Status 404 returned error can't find the container with id 231f5101024a82c58e876a4b91cb2af968971b93dff119c379a7825a85a77c88
Oct 30 05:34:35.731707 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:35.731674    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6" event={"ID":"115b8606-cbec-4642-9dc9-61abefa440af","Type":"ContainerStarted","Data":"231f5101024a82c58e876a4b91cb2af968971b93dff119c379a7825a85a77c88"}
Oct 30 05:34:36.738975 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:36.738936    5072 generic.go:356] "Generic (PLEG): container finished" podID="115b8606-cbec-4642-9dc9-61abefa440af" containerID="8d3dc7398c5e56f067e138b4c25eb16c92b0716d3940bc450d524fe28726372f" exitCode=0
Oct 30 05:34:36.739465 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:36.739029    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6" event={"ID":"115b8606-cbec-4642-9dc9-61abefa440af","Type":"ContainerDied","Data":"8d3dc7398c5e56f067e138b4c25eb16c92b0716d3940bc450d524fe28726372f"}
Oct 30 05:34:37.890574 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:37.890552    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:38.083365 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:38.083285    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115b8606-cbec-4642-9dc9-61abefa440af-kube-api-access-z8t2j\" (UniqueName: \"kubernetes.io/projected/115b8606-cbec-4642-9dc9-61abefa440af-kube-api-access-z8t2j\") pod \"115b8606-cbec-4642-9dc9-61abefa440af\" (UID: \"115b8606-cbec-4642-9dc9-61abefa440af\") "
Oct 30 05:34:38.083365 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:38.083324    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-xks-ca\") pod \"115b8606-cbec-4642-9dc9-61abefa440af\" (UID: \"115b8606-cbec-4642-9dc9-61abefa440af\") "
Oct 30 05:34:38.083616 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:38.083504    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-openshift-ca\") pod \"115b8606-cbec-4642-9dc9-61abefa440af\" (UID: \"115b8606-cbec-4642-9dc9-61abefa440af\") "
Oct 30 05:34:38.083840 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:38.083815    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-openshift-ca" pod "115b8606-cbec-4642-9dc9-61abefa440af" (UID: "115b8606-cbec-4642-9dc9-61abefa440af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:34:38.085568 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:38.085545    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-xks-ca" pod "115b8606-cbec-4642-9dc9-61abefa440af" (UID: "115b8606-cbec-4642-9dc9-61abefa440af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:34:38.085721 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:38.085697    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115b8606-cbec-4642-9dc9-61abefa440af-kube-api-access-z8t2j" pod "115b8606-cbec-4642-9dc9-61abefa440af" (UID: "115b8606-cbec-4642-9dc9-61abefa440af"). InnerVolumeSpecName "kube-api-access-z8t2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:34:38.184527 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:38.184492    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:34:38.184527 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:38.184521    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8t2j\" (UniqueName: \"kubernetes.io/projected/115b8606-cbec-4642-9dc9-61abefa440af-kube-api-access-z8t2j\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:34:38.184682 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:38.184536    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115b8606-cbec-4642-9dc9-61abefa440af-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:34:38.751458 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:38.751426    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6" event={"ID":"115b8606-cbec-4642-9dc9-61abefa440af","Type":"ContainerDied","Data":"231f5101024a82c58e876a4b91cb2af968971b93dff119c379a7825a85a77c88"}
Oct 30 05:34:38.751651 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:38.751470    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="231f5101024a82c58e876a4b91cb2af968971b93dff119c379a7825a85a77c88"
Oct 30 05:34:38.751651 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:38.751468    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4ed2e9-wszd6"
Oct 30 05:34:41.750648 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:34:41.750614    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:34:53.811114 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:53.810220    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-smfn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:34:53.811114 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:53.810335    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-smfn4" podUID="ac611251-3890-4b60-b3a3-fb444a6befbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:34:56.995260 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:56.995215    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-txlk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:34:56.995798 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:34:56.995281    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4" podUID="b748c4e2-2e95-43e6-8dc4-9bf3958a24d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:35:06.256388 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.256350    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"]
Oct 30 05:35:06.257206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.257185    5072 cpu_manager.go:414] "containerMap: removing container" podUID="115b8606-cbec-4642-9dc9-61abefa440af" containerName="revoke-keys"
Oct 30 05:35:06.257265 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.257209    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115b8606-cbec-4642-9dc9-61abefa440af" containerName="revoke-keys"
Oct 30 05:35:06.257345 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.257333    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="115b8606-cbec-4642-9dc9-61abefa440af" containerName="revoke-keys"
Oct 30 05:35:06.261326 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.261298    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:06.277458 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.277433    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"]
Oct 30 05:35:06.287475 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.287450    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:35:06.287649 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.287626    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:35:06.287794 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.287772    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:35:06.287971 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.287857    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:35:06.430160 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.430123    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e42eb7ab-pv7pd\" (UID: \"56f62958-7c41-45f5-83b2-83abfebe5bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:06.430352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.430237    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td6qz\" (UniqueName: \"kubernetes.io/projected/56f62958-7c41-45f5-83b2-83abfebe5bca-kube-api-access-td6qz\") pod \"maas-api-revoke-sub-e42eb7ab-pv7pd\" (UID: \"56f62958-7c41-45f5-83b2-83abfebe5bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:06.430352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.430286    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e42eb7ab-pv7pd\" (UID: \"56f62958-7c41-45f5-83b2-83abfebe5bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:06.530929 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.530837    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e42eb7ab-pv7pd\" (UID: \"56f62958-7c41-45f5-83b2-83abfebe5bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:06.531174 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.531148    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td6qz\" (UniqueName: \"kubernetes.io/projected/56f62958-7c41-45f5-83b2-83abfebe5bca-kube-api-access-td6qz\") pod \"maas-api-revoke-sub-e42eb7ab-pv7pd\" (UID: \"56f62958-7c41-45f5-83b2-83abfebe5bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:06.531269 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.531228    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e42eb7ab-pv7pd\" (UID: \"56f62958-7c41-45f5-83b2-83abfebe5bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:06.531605 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.531581    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e42eb7ab-pv7pd\" (UID: \"56f62958-7c41-45f5-83b2-83abfebe5bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:06.533737 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.533716    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e42eb7ab-pv7pd\" (UID: \"56f62958-7c41-45f5-83b2-83abfebe5bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:06.555179 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.555154    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td6qz\" (UniqueName: \"kubernetes.io/projected/56f62958-7c41-45f5-83b2-83abfebe5bca-kube-api-access-td6qz\") pod \"maas-api-revoke-sub-e42eb7ab-pv7pd\" (UID: \"56f62958-7c41-45f5-83b2-83abfebe5bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:06.575142 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.575121    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:06.875568 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.875541    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"]
Oct 30 05:35:06.876463 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:35:06.876436    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f62958_7c41_45f5_83b2_83abfebe5bca.slice/crio-7d76d7a4b08cb0a3eb5c5347ad38af96eda4396c8fc53e4e2e5f5f41ffeeab17 WatchSource:0}: Error finding container 7d76d7a4b08cb0a3eb5c5347ad38af96eda4396c8fc53e4e2e5f5f41ffeeab17: Status 404 returned error can't find the container with id 7d76d7a4b08cb0a3eb5c5347ad38af96eda4396c8fc53e4e2e5f5f41ffeeab17
Oct 30 05:35:06.952682 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:06.952644    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd" event={"ID":"56f62958-7c41-45f5-83b2-83abfebe5bca","Type":"ContainerStarted","Data":"7d76d7a4b08cb0a3eb5c5347ad38af96eda4396c8fc53e4e2e5f5f41ffeeab17"}
Oct 30 05:35:07.959159 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:07.959128    5072 generic.go:356] "Generic (PLEG): container finished" podID="56f62958-7c41-45f5-83b2-83abfebe5bca" containerID="ca851ea1ca618178109b30c5e558089b9c7f54e1d12f5990e0ad11a745dc8268" exitCode=0
Oct 30 05:35:07.959552 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:07.959220    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd" event={"ID":"56f62958-7c41-45f5-83b2-83abfebe5bca","Type":"ContainerDied","Data":"ca851ea1ca618178109b30c5e558089b9c7f54e1d12f5990e0ad11a745dc8268"}
Oct 30 05:35:09.106815 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.106789    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:09.258937 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.258869    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-openshift-ca\") pod \"56f62958-7c41-45f5-83b2-83abfebe5bca\" (UID: \"56f62958-7c41-45f5-83b2-83abfebe5bca\") "
Oct 30 05:35:09.259100 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.258947    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-xks-ca\") pod \"56f62958-7c41-45f5-83b2-83abfebe5bca\" (UID: \"56f62958-7c41-45f5-83b2-83abfebe5bca\") "
Oct 30 05:35:09.259100 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.259032    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f62958-7c41-45f5-83b2-83abfebe5bca-kube-api-access-td6qz\" (UniqueName: \"kubernetes.io/projected/56f62958-7c41-45f5-83b2-83abfebe5bca-kube-api-access-td6qz\") pod \"56f62958-7c41-45f5-83b2-83abfebe5bca\" (UID: \"56f62958-7c41-45f5-83b2-83abfebe5bca\") "
Oct 30 05:35:09.259222 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.259202    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-openshift-ca" pod "56f62958-7c41-45f5-83b2-83abfebe5bca" (UID: "56f62958-7c41-45f5-83b2-83abfebe5bca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:35:09.259410 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.259387    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:35:09.261173 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.261148    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-xks-ca" pod "56f62958-7c41-45f5-83b2-83abfebe5bca" (UID: "56f62958-7c41-45f5-83b2-83abfebe5bca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:35:09.261267 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.261235    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f62958-7c41-45f5-83b2-83abfebe5bca-kube-api-access-td6qz" pod "56f62958-7c41-45f5-83b2-83abfebe5bca" (UID: "56f62958-7c41-45f5-83b2-83abfebe5bca"). InnerVolumeSpecName "kube-api-access-td6qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:09.360303 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.360270    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f62958-7c41-45f5-83b2-83abfebe5bca-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:35:09.360303 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.360299    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td6qz\" (UniqueName: \"kubernetes.io/projected/56f62958-7c41-45f5-83b2-83abfebe5bca-kube-api-access-td6qz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:35:09.970601 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.970574    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd"
Oct 30 05:35:09.970778 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.970573    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42eb7ab-pv7pd" event={"ID":"56f62958-7c41-45f5-83b2-83abfebe5bca","Type":"ContainerDied","Data":"7d76d7a4b08cb0a3eb5c5347ad38af96eda4396c8fc53e4e2e5f5f41ffeeab17"}
Oct 30 05:35:09.970778 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:09.970690    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d76d7a4b08cb0a3eb5c5347ad38af96eda4396c8fc53e4e2e5f5f41ffeeab17"
Oct 30 05:35:11.751601 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:35:11.751557    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:35:16.608453 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.608409    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"]
Oct 30 05:35:16.608966 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.608950    5072 cpu_manager.go:414] "containerMap: removing container" podUID="56f62958-7c41-45f5-83b2-83abfebe5bca" containerName="revoke-keys"
Oct 30 05:35:16.608966 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.608967    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f62958-7c41-45f5-83b2-83abfebe5bca" containerName="revoke-keys"
Oct 30 05:35:16.609058 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.609049    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f62958-7c41-45f5-83b2-83abfebe5bca" containerName="revoke-keys"
Oct 30 05:35:16.612533 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.612512    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:16.628101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.628049    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:35:16.630736 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.630709    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"]
Oct 30 05:35:16.640863 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.640842    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:35:16.641118 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.641094    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:35:16.641118 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.641108    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:35:16.732149 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.732119    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlvt9\" (UniqueName: \"kubernetes.io/projected/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-kube-api-access-qlvt9\") pod \"maas-api-revoke-sub-f5d6c54a-xtjnr\" (UID: \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:16.732317 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.732182    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5d6c54a-xtjnr\" (UID: \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:16.732367 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.732326    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5d6c54a-xtjnr\" (UID: \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:16.832809 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.832785    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlvt9\" (UniqueName: \"kubernetes.io/projected/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-kube-api-access-qlvt9\") pod \"maas-api-revoke-sub-f5d6c54a-xtjnr\" (UID: \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:16.832975 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.832838    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5d6c54a-xtjnr\" (UID: \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:16.832975 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.832889    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5d6c54a-xtjnr\" (UID: \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:16.833777 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.833748    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5d6c54a-xtjnr\" (UID: \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:16.835411 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.835387    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5d6c54a-xtjnr\" (UID: \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:16.857569 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.857546    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlvt9\" (UniqueName: \"kubernetes.io/projected/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-kube-api-access-qlvt9\") pod \"maas-api-revoke-sub-f5d6c54a-xtjnr\" (UID: \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:16.932263 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:16.932208    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:17.233402 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:17.233364    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"]
Oct 30 05:35:17.234464 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:35:17.234437    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda111c0cb_6dc1_40f9_aa51_c0a861c305aa.slice/crio-726c177da0ccf7a3aca4939eedbebadc9d7a794ce30689e4416a64611049246d WatchSource:0}: Error finding container 726c177da0ccf7a3aca4939eedbebadc9d7a794ce30689e4416a64611049246d: Status 404 returned error can't find the container with id 726c177da0ccf7a3aca4939eedbebadc9d7a794ce30689e4416a64611049246d
Oct 30 05:35:18.015744 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:18.015701    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr" event={"ID":"a111c0cb-6dc1-40f9-aa51-c0a861c305aa","Type":"ContainerStarted","Data":"3c73c3b7bc7fdf174b517fbefd50f92dceaefe270050cc5541810043c0efd914"}
Oct 30 05:35:18.016214 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:18.015764    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr" event={"ID":"a111c0cb-6dc1-40f9-aa51-c0a861c305aa","Type":"ContainerStarted","Data":"726c177da0ccf7a3aca4939eedbebadc9d7a794ce30689e4416a64611049246d"}
Oct 30 05:35:18.065141 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:18.065062    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr" podStartSLOduration=2.065042224 podStartE2EDuration="2.065042224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:35:18.064006385 +0000 UTC m=+5897.805429483" watchObservedRunningTime="2026-10-30 05:35:18.065042224 +0000 UTC m=+5897.806465324"
Oct 30 05:35:19.030110 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:19.029580    5072 generic.go:356] "Generic (PLEG): container finished" podID="a111c0cb-6dc1-40f9-aa51-c0a861c305aa" containerID="3c73c3b7bc7fdf174b517fbefd50f92dceaefe270050cc5541810043c0efd914" exitCode=0
Oct 30 05:35:19.030110 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:19.029697    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr" event={"ID":"a111c0cb-6dc1-40f9-aa51-c0a861c305aa","Type":"ContainerDied","Data":"3c73c3b7bc7fdf174b517fbefd50f92dceaefe270050cc5541810043c0efd914"}
Oct 30 05:35:20.386496 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:20.386196    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gl57g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:35:20.386496 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:20.386258    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gl57g" podUID="8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:35:20.436103 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:20.436044    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:20.535296 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:20.535264    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-openshift-ca\") pod \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\" (UID: \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\") "
Oct 30 05:35:20.535565 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:20.535545    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-kube-api-access-qlvt9\" (UniqueName: \"kubernetes.io/projected/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-kube-api-access-qlvt9\") pod \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\" (UID: \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\") "
Oct 30 05:35:20.535665 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:20.535610    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-xks-ca\") pod \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\" (UID: \"a111c0cb-6dc1-40f9-aa51-c0a861c305aa\") "
Oct 30 05:35:20.535837 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:20.535756    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-openshift-ca" pod "a111c0cb-6dc1-40f9-aa51-c0a861c305aa" (UID: "a111c0cb-6dc1-40f9-aa51-c0a861c305aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:35:20.536310 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:20.536270    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:35:20.538827 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:20.538779    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-xks-ca" pod "a111c0cb-6dc1-40f9-aa51-c0a861c305aa" (UID: "a111c0cb-6dc1-40f9-aa51-c0a861c305aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:35:20.538827 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:20.538781    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-kube-api-access-qlvt9" pod "a111c0cb-6dc1-40f9-aa51-c0a861c305aa" (UID: "a111c0cb-6dc1-40f9-aa51-c0a861c305aa"). InnerVolumeSpecName "kube-api-access-qlvt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:20.637513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:20.637430    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlvt9\" (UniqueName: \"kubernetes.io/projected/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-kube-api-access-qlvt9\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:35:20.637513 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:20.637468    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a111c0cb-6dc1-40f9-aa51-c0a861c305aa-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:35:21.044177 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:21.044147    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr"
Oct 30 05:35:21.044353 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:21.044179    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5d6c54a-xtjnr" event={"ID":"a111c0cb-6dc1-40f9-aa51-c0a861c305aa","Type":"ContainerDied","Data":"726c177da0ccf7a3aca4939eedbebadc9d7a794ce30689e4416a64611049246d"}
Oct 30 05:35:21.044353 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:21.044229    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="726c177da0ccf7a3aca4939eedbebadc9d7a794ce30689e4416a64611049246d"
Oct 30 05:35:41.751629 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:35:41.751589    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:35:49.346105 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.346056    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"]
Oct 30 05:35:49.346608 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.346577    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a111c0cb-6dc1-40f9-aa51-c0a861c305aa" containerName="revoke-keys"
Oct 30 05:35:49.346608 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.346588    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a111c0cb-6dc1-40f9-aa51-c0a861c305aa" containerName="revoke-keys"
Oct 30 05:35:49.346702 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.346691    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a111c0cb-6dc1-40f9-aa51-c0a861c305aa" containerName="revoke-keys"
Oct 30 05:35:49.349925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.349907    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:49.384571 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.384540    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:35:49.384766 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.384709    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:35:49.384766 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.384709    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:35:49.385059 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.385025    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:35:49.388043 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.388022    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"]
Oct 30 05:35:49.470801 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.470759    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23468440-kxzj4\" (UID: \"b6e86583-d955-43ef-a522-89aac23f0526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:49.471039 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.470839    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23468440-kxzj4\" (UID: \"b6e86583-d955-43ef-a522-89aac23f0526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:49.471039 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.470865    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7c98\" (UniqueName: \"kubernetes.io/projected/b6e86583-d955-43ef-a522-89aac23f0526-kube-api-access-k7c98\") pod \"maas-api-revoke-sub-23468440-kxzj4\" (UID: \"b6e86583-d955-43ef-a522-89aac23f0526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:49.572003 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.571963    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23468440-kxzj4\" (UID: \"b6e86583-d955-43ef-a522-89aac23f0526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:49.572265 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.572013    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7c98\" (UniqueName: \"kubernetes.io/projected/b6e86583-d955-43ef-a522-89aac23f0526-kube-api-access-k7c98\") pod \"maas-api-revoke-sub-23468440-kxzj4\" (UID: \"b6e86583-d955-43ef-a522-89aac23f0526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:49.572265 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.572210    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23468440-kxzj4\" (UID: \"b6e86583-d955-43ef-a522-89aac23f0526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:49.572826 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.572804    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23468440-kxzj4\" (UID: \"b6e86583-d955-43ef-a522-89aac23f0526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:49.574720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.574690    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23468440-kxzj4\" (UID: \"b6e86583-d955-43ef-a522-89aac23f0526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:49.600041 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.599982    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7c98\" (UniqueName: \"kubernetes.io/projected/b6e86583-d955-43ef-a522-89aac23f0526-kube-api-access-k7c98\") pod \"maas-api-revoke-sub-23468440-kxzj4\" (UID: \"b6e86583-d955-43ef-a522-89aac23f0526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:49.663797 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.663771    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:49.972386 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:49.972356    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"]
Oct 30 05:35:49.972508 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:35:49.972369    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e86583_d955_43ef_a522_89aac23f0526.slice/crio-73b277d0b464865934ab9578613649a408443d2c5fdb365d4501b36e45ac83b7 WatchSource:0}: Error finding container 73b277d0b464865934ab9578613649a408443d2c5fdb365d4501b36e45ac83b7: Status 404 returned error can't find the container with id 73b277d0b464865934ab9578613649a408443d2c5fdb365d4501b36e45ac83b7
Oct 30 05:35:50.255258 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:50.255187    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4" event={"ID":"b6e86583-d955-43ef-a522-89aac23f0526","Type":"ContainerStarted","Data":"73b277d0b464865934ab9578613649a408443d2c5fdb365d4501b36e45ac83b7"}
Oct 30 05:35:51.261996 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:51.261959    5072 generic.go:356] "Generic (PLEG): container finished" podID="b6e86583-d955-43ef-a522-89aac23f0526" containerID="75400f3a14cab88c629cc52d1f5c8e781472555c0ff4cb603058ea6ca707c271" exitCode=0
Oct 30 05:35:51.262407 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:51.262027    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4" event={"ID":"b6e86583-d955-43ef-a522-89aac23f0526","Type":"ContainerDied","Data":"75400f3a14cab88c629cc52d1f5c8e781472555c0ff4cb603058ea6ca707c271"}
Oct 30 05:35:52.417421 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:52.417397    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:52.500194 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:52.500162    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-xks-ca\") pod \"b6e86583-d955-43ef-a522-89aac23f0526\" (UID: \"b6e86583-d955-43ef-a522-89aac23f0526\") "
Oct 30 05:35:52.500332 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:52.500248    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e86583-d955-43ef-a522-89aac23f0526-kube-api-access-k7c98\" (UniqueName: \"kubernetes.io/projected/b6e86583-d955-43ef-a522-89aac23f0526-kube-api-access-k7c98\") pod \"b6e86583-d955-43ef-a522-89aac23f0526\" (UID: \"b6e86583-d955-43ef-a522-89aac23f0526\") "
Oct 30 05:35:52.500332 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:52.500289    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-openshift-ca\") pod \"b6e86583-d955-43ef-a522-89aac23f0526\" (UID: \"b6e86583-d955-43ef-a522-89aac23f0526\") "
Oct 30 05:35:52.500682 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:52.500658    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-openshift-ca" pod "b6e86583-d955-43ef-a522-89aac23f0526" (UID: "b6e86583-d955-43ef-a522-89aac23f0526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:35:52.502480 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:52.502453    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e86583-d955-43ef-a522-89aac23f0526-kube-api-access-k7c98" pod "b6e86583-d955-43ef-a522-89aac23f0526" (UID: "b6e86583-d955-43ef-a522-89aac23f0526"). InnerVolumeSpecName "kube-api-access-k7c98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:52.502480 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:52.502469    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-xks-ca" pod "b6e86583-d955-43ef-a522-89aac23f0526" (UID: "b6e86583-d955-43ef-a522-89aac23f0526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:35:52.601882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:52.601812    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7c98\" (UniqueName: \"kubernetes.io/projected/b6e86583-d955-43ef-a522-89aac23f0526-kube-api-access-k7c98\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:35:52.601882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:52.601839    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:35:52.601882 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:52.601849    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e86583-d955-43ef-a522-89aac23f0526-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:35:53.274367 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:53.274335    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4"
Oct 30 05:35:53.274547 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:53.274364    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23468440-kxzj4" event={"ID":"b6e86583-d955-43ef-a522-89aac23f0526","Type":"ContainerDied","Data":"73b277d0b464865934ab9578613649a408443d2c5fdb365d4501b36e45ac83b7"}
Oct 30 05:35:53.274547 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:53.274410    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b277d0b464865934ab9578613649a408443d2c5fdb365d4501b36e45ac83b7"
Oct 30 05:35:59.779813 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.779781    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"]
Oct 30 05:35:59.780316 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.780292    5072 cpu_manager.go:414] "containerMap: removing container" podUID="b6e86583-d955-43ef-a522-89aac23f0526" containerName="revoke-keys"
Oct 30 05:35:59.780316 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.780306    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e86583-d955-43ef-a522-89aac23f0526" containerName="revoke-keys"
Oct 30 05:35:59.780410 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.780399    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e86583-d955-43ef-a522-89aac23f0526" containerName="revoke-keys"
Oct 30 05:35:59.783766 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.783746    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:35:59.803504 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.803485    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"]
Oct 30 05:35:59.813186 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.813161    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:35:59.813386 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.813360    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:35:59.813386 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.813269    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:35:59.816232 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.816210    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:35:59.973098 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.973044    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fdc02eb-4gncp\" (UID: \"6f44dc83-903b-403e-ad00-049dac8b7636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:35:59.973264 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.973108    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fdc02eb-4gncp\" (UID: \"6f44dc83-903b-403e-ad00-049dac8b7636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:35:59.973264 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:35:59.973200    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ktbv\" (UniqueName: \"kubernetes.io/projected/6f44dc83-903b-403e-ad00-049dac8b7636-kube-api-access-4ktbv\") pod \"maas-api-revoke-sub-4fdc02eb-4gncp\" (UID: \"6f44dc83-903b-403e-ad00-049dac8b7636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:36:00.073852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:00.073769    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fdc02eb-4gncp\" (UID: \"6f44dc83-903b-403e-ad00-049dac8b7636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:36:00.073852 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:00.073822    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ktbv\" (UniqueName: \"kubernetes.io/projected/6f44dc83-903b-403e-ad00-049dac8b7636-kube-api-access-4ktbv\") pod \"maas-api-revoke-sub-4fdc02eb-4gncp\" (UID: \"6f44dc83-903b-403e-ad00-049dac8b7636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:36:00.074109 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:00.073921    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fdc02eb-4gncp\" (UID: \"6f44dc83-903b-403e-ad00-049dac8b7636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:36:00.074669 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:00.074635    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fdc02eb-4gncp\" (UID: \"6f44dc83-903b-403e-ad00-049dac8b7636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:36:00.076596 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:00.076574    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fdc02eb-4gncp\" (UID: \"6f44dc83-903b-403e-ad00-049dac8b7636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:36:00.098755 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:00.098731    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ktbv\" (UniqueName: \"kubernetes.io/projected/6f44dc83-903b-403e-ad00-049dac8b7636-kube-api-access-4ktbv\") pod \"maas-api-revoke-sub-4fdc02eb-4gncp\" (UID: \"6f44dc83-903b-403e-ad00-049dac8b7636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:36:00.396719 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:00.396638    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:36:00.701204 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:00.701172    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"]
Oct 30 05:36:00.702174 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:36:00.702142    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f44dc83_903b_403e_ad00_049dac8b7636.slice/crio-2957d9a46c0d0e08e8f8aac0a4abd8a27e062410f7200e3bc575c632a4d292c1 WatchSource:0}: Error finding container 2957d9a46c0d0e08e8f8aac0a4abd8a27e062410f7200e3bc575c632a4d292c1: Status 404 returned error can't find the container with id 2957d9a46c0d0e08e8f8aac0a4abd8a27e062410f7200e3bc575c632a4d292c1
Oct 30 05:36:01.319731 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:01.319638    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp" event={"ID":"6f44dc83-903b-403e-ad00-049dac8b7636","Type":"ContainerStarted","Data":"e1f8b0f23f76e74c1f1ee473329b00ccbecdc13fec76d59e6cd8646316677dc1"}
Oct 30 05:36:01.319731 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:01.319695    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp" event={"ID":"6f44dc83-903b-403e-ad00-049dac8b7636","Type":"ContainerStarted","Data":"2957d9a46c0d0e08e8f8aac0a4abd8a27e062410f7200e3bc575c632a4d292c1"}
Oct 30 05:36:02.328869 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:02.328826    5072 generic.go:356] "Generic (PLEG): container finished" podID="6f44dc83-903b-403e-ad00-049dac8b7636" containerID="e1f8b0f23f76e74c1f1ee473329b00ccbecdc13fec76d59e6cd8646316677dc1" exitCode=0
Oct 30 05:36:02.329441 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:02.328908    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp" event={"ID":"6f44dc83-903b-403e-ad00-049dac8b7636","Type":"ContainerDied","Data":"e1f8b0f23f76e74c1f1ee473329b00ccbecdc13fec76d59e6cd8646316677dc1"}
Oct 30 05:36:03.728167 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:03.728012    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:36:03.741849 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:03.741816    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-xks-ca\") pod \"6f44dc83-903b-403e-ad00-049dac8b7636\" (UID: \"6f44dc83-903b-403e-ad00-049dac8b7636\") "
Oct 30 05:36:03.741981 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:03.741868    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-openshift-ca\") pod \"6f44dc83-903b-403e-ad00-049dac8b7636\" (UID: \"6f44dc83-903b-403e-ad00-049dac8b7636\") "
Oct 30 05:36:03.742124 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:03.742041    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f44dc83-903b-403e-ad00-049dac8b7636-kube-api-access-4ktbv\" (UniqueName: \"kubernetes.io/projected/6f44dc83-903b-403e-ad00-049dac8b7636-kube-api-access-4ktbv\") pod \"6f44dc83-903b-403e-ad00-049dac8b7636\" (UID: \"6f44dc83-903b-403e-ad00-049dac8b7636\") "
Oct 30 05:36:03.742386 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:03.742362    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-openshift-ca" pod "6f44dc83-903b-403e-ad00-049dac8b7636" (UID: "6f44dc83-903b-403e-ad00-049dac8b7636"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:36:03.742508 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:03.742494    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:03.744247 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:03.744222    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f44dc83-903b-403e-ad00-049dac8b7636-kube-api-access-4ktbv" pod "6f44dc83-903b-403e-ad00-049dac8b7636" (UID: "6f44dc83-903b-403e-ad00-049dac8b7636"). InnerVolumeSpecName "kube-api-access-4ktbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:03.744360 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:03.744285    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-xks-ca" pod "6f44dc83-903b-403e-ad00-049dac8b7636" (UID: "6f44dc83-903b-403e-ad00-049dac8b7636"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:36:03.843767 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:03.843714    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ktbv\" (UniqueName: \"kubernetes.io/projected/6f44dc83-903b-403e-ad00-049dac8b7636-kube-api-access-4ktbv\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:03.843767 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:03.843744    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f44dc83-903b-403e-ad00-049dac8b7636-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:04.344063 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:04.344025    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp"
Oct 30 05:36:04.344063 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:04.344057    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc02eb-4gncp" event={"ID":"6f44dc83-903b-403e-ad00-049dac8b7636","Type":"ContainerDied","Data":"2957d9a46c0d0e08e8f8aac0a4abd8a27e062410f7200e3bc575c632a4d292c1"}
Oct 30 05:36:04.344307 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:04.344135    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2957d9a46c0d0e08e8f8aac0a4abd8a27e062410f7200e3bc575c632a4d292c1"
Oct 30 05:36:10.120519 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.120483    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"]
Oct 30 05:36:10.122352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.122323    5072 cpu_manager.go:414] "containerMap: removing container" podUID="6f44dc83-903b-403e-ad00-049dac8b7636" containerName="revoke-keys"
Oct 30 05:36:10.122352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.122351    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f44dc83-903b-403e-ad00-049dac8b7636" containerName="revoke-keys"
Oct 30 05:36:10.122660 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.122640    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f44dc83-903b-403e-ad00-049dac8b7636" containerName="revoke-keys"
Oct 30 05:36:10.128261 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.128223    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:10.137525 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.137493    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"]
Oct 30 05:36:10.146331 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.146308    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:36:10.146563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.146546    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:36:10.146679 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.146314    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:36:10.146892 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.146862    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:36:10.207685 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.207651    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e691efff-dz2ml\" (UID: \"a6a28035-dce5-45fe-8a79-694a5c2ea708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:10.207895 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.207689    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j476d\" (UniqueName: \"kubernetes.io/projected/a6a28035-dce5-45fe-8a79-694a5c2ea708-kube-api-access-j476d\") pod \"maas-api-revoke-sub-e691efff-dz2ml\" (UID: \"a6a28035-dce5-45fe-8a79-694a5c2ea708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:10.207895 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.207826    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e691efff-dz2ml\" (UID: \"a6a28035-dce5-45fe-8a79-694a5c2ea708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:10.308463 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.308425    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e691efff-dz2ml\" (UID: \"a6a28035-dce5-45fe-8a79-694a5c2ea708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:10.308629 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.308496    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e691efff-dz2ml\" (UID: \"a6a28035-dce5-45fe-8a79-694a5c2ea708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:10.308629 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.308535    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j476d\" (UniqueName: \"kubernetes.io/projected/a6a28035-dce5-45fe-8a79-694a5c2ea708-kube-api-access-j476d\") pod \"maas-api-revoke-sub-e691efff-dz2ml\" (UID: \"a6a28035-dce5-45fe-8a79-694a5c2ea708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:10.309066 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.309037    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e691efff-dz2ml\" (UID: \"a6a28035-dce5-45fe-8a79-694a5c2ea708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:10.311025 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.311000    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e691efff-dz2ml\" (UID: \"a6a28035-dce5-45fe-8a79-694a5c2ea708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:10.332459 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.332428    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j476d\" (UniqueName: \"kubernetes.io/projected/a6a28035-dce5-45fe-8a79-694a5c2ea708-kube-api-access-j476d\") pod \"maas-api-revoke-sub-e691efff-dz2ml\" (UID: \"a6a28035-dce5-45fe-8a79-694a5c2ea708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:10.449860 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.449828    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:10.758865 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:36:10.758826    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a28035_dce5_45fe_8a79_694a5c2ea708.slice/crio-3a38d0886909c205094232b83b871fe827ab8d46a0f7c72e52a906ca4105d372 WatchSource:0}: Error finding container 3a38d0886909c205094232b83b871fe827ab8d46a0f7c72e52a906ca4105d372: Status 404 returned error can't find the container with id 3a38d0886909c205094232b83b871fe827ab8d46a0f7c72e52a906ca4105d372
Oct 30 05:36:10.759580 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:10.759550    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"]
Oct 30 05:36:11.384805 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:11.384718    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml" event={"ID":"a6a28035-dce5-45fe-8a79-694a5c2ea708","Type":"ContainerStarted","Data":"6ff7d429e72692c85489d3d54ae647515eb7f0f19642987b9288852120b8d83f"}
Oct 30 05:36:11.384805 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:11.384779    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml" event={"ID":"a6a28035-dce5-45fe-8a79-694a5c2ea708","Type":"ContainerStarted","Data":"3a38d0886909c205094232b83b871fe827ab8d46a0f7c72e52a906ca4105d372"}
Oct 30 05:36:11.427253 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:11.427204    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml" podStartSLOduration=1.427191213 podStartE2EDuration="1.427191213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:11.425810155 +0000 UTC m=+5951.167233252" watchObservedRunningTime="2026-10-30 05:36:11.427191213 +0000 UTC m=+5951.168614310"
Oct 30 05:36:11.750307 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:36:11.750275    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:36:12.391792 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:12.391756    5072 generic.go:356] "Generic (PLEG): container finished" podID="a6a28035-dce5-45fe-8a79-694a5c2ea708" containerID="6ff7d429e72692c85489d3d54ae647515eb7f0f19642987b9288852120b8d83f" exitCode=0
Oct 30 05:36:12.392287 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:12.391846    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml" event={"ID":"a6a28035-dce5-45fe-8a79-694a5c2ea708","Type":"ContainerDied","Data":"6ff7d429e72692c85489d3d54ae647515eb7f0f19642987b9288852120b8d83f"}
Oct 30 05:36:13.539433 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:13.539401    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:13.640624 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:13.640589    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a28035-dce5-45fe-8a79-694a5c2ea708-kube-api-access-j476d\" (UniqueName: \"kubernetes.io/projected/a6a28035-dce5-45fe-8a79-694a5c2ea708-kube-api-access-j476d\") pod \"a6a28035-dce5-45fe-8a79-694a5c2ea708\" (UID: \"a6a28035-dce5-45fe-8a79-694a5c2ea708\") "
Oct 30 05:36:13.640777 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:13.640637    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-openshift-ca\") pod \"a6a28035-dce5-45fe-8a79-694a5c2ea708\" (UID: \"a6a28035-dce5-45fe-8a79-694a5c2ea708\") "
Oct 30 05:36:13.640777 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:13.640683    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-xks-ca\") pod \"a6a28035-dce5-45fe-8a79-694a5c2ea708\" (UID: \"a6a28035-dce5-45fe-8a79-694a5c2ea708\") "
Oct 30 05:36:13.641041 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:13.641016    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-openshift-ca" pod "a6a28035-dce5-45fe-8a79-694a5c2ea708" (UID: "a6a28035-dce5-45fe-8a79-694a5c2ea708"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:36:13.642973 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:13.642948    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-xks-ca" pod "a6a28035-dce5-45fe-8a79-694a5c2ea708" (UID: "a6a28035-dce5-45fe-8a79-694a5c2ea708"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:36:13.642973 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:13.642959    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a28035-dce5-45fe-8a79-694a5c2ea708-kube-api-access-j476d" pod "a6a28035-dce5-45fe-8a79-694a5c2ea708" (UID: "a6a28035-dce5-45fe-8a79-694a5c2ea708"). InnerVolumeSpecName "kube-api-access-j476d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:13.741597 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:13.741561    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j476d\" (UniqueName: \"kubernetes.io/projected/a6a28035-dce5-45fe-8a79-694a5c2ea708-kube-api-access-j476d\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:13.741597 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:13.741593    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:13.741795 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:13.741607    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a28035-dce5-45fe-8a79-694a5c2ea708-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:14.403907 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:14.403876    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml"
Oct 30 05:36:14.406496 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:14.406469    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e691efff-dz2ml" event={"ID":"a6a28035-dce5-45fe-8a79-694a5c2ea708","Type":"ContainerDied","Data":"3a38d0886909c205094232b83b871fe827ab8d46a0f7c72e52a906ca4105d372"}
Oct 30 05:36:14.406646 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:14.406509    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a38d0886909c205094232b83b871fe827ab8d46a0f7c72e52a906ca4105d372"
Oct 30 05:36:20.586801 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.586771    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"]
Oct 30 05:36:20.587339 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.587319    5072 cpu_manager.go:414] "containerMap: removing container" podUID="a6a28035-dce5-45fe-8a79-694a5c2ea708" containerName="revoke-keys"
Oct 30 05:36:20.587390 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.587340    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a28035-dce5-45fe-8a79-694a5c2ea708" containerName="revoke-keys"
Oct 30 05:36:20.587440 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.587431    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a28035-dce5-45fe-8a79-694a5c2ea708" containerName="revoke-keys"
Oct 30 05:36:20.590645 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.590630    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:20.606298 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.606252    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:36:20.619011 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.618982    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:36:20.619156 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.618994    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:36:20.619156 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.619027    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:36:20.621921 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.621899    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"]
Oct 30 05:36:20.707025 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.706987    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08dbc435-2ccbv\" (UID: \"3b61e548-7a4e-4c51-b45d-9f87de504c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:20.707206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.707029    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08dbc435-2ccbv\" (UID: \"3b61e548-7a4e-4c51-b45d-9f87de504c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:20.707264 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.707246    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcrwz\" (UniqueName: \"kubernetes.io/projected/3b61e548-7a4e-4c51-b45d-9f87de504c48-kube-api-access-kcrwz\") pod \"maas-api-revoke-sub-08dbc435-2ccbv\" (UID: \"3b61e548-7a4e-4c51-b45d-9f87de504c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:20.808608 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.808578    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcrwz\" (UniqueName: \"kubernetes.io/projected/3b61e548-7a4e-4c51-b45d-9f87de504c48-kube-api-access-kcrwz\") pod \"maas-api-revoke-sub-08dbc435-2ccbv\" (UID: \"3b61e548-7a4e-4c51-b45d-9f87de504c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:20.808746 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.808651    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08dbc435-2ccbv\" (UID: \"3b61e548-7a4e-4c51-b45d-9f87de504c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:20.808746 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.808681    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08dbc435-2ccbv\" (UID: \"3b61e548-7a4e-4c51-b45d-9f87de504c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:20.809312 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.809294    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08dbc435-2ccbv\" (UID: \"3b61e548-7a4e-4c51-b45d-9f87de504c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:20.811222 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.811204    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08dbc435-2ccbv\" (UID: \"3b61e548-7a4e-4c51-b45d-9f87de504c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:20.832715 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.832687    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcrwz\" (UniqueName: \"kubernetes.io/projected/3b61e548-7a4e-4c51-b45d-9f87de504c48-kube-api-access-kcrwz\") pod \"maas-api-revoke-sub-08dbc435-2ccbv\" (UID: \"3b61e548-7a4e-4c51-b45d-9f87de504c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:20.903804 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:20.903745    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:21.205194 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:36:21.205162    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b61e548_7a4e_4c51_b45d_9f87de504c48.slice/crio-e8681336d6796619f30aeea6f4081711af8dc4481db748955ef288b66f89f0f4 WatchSource:0}: Error finding container e8681336d6796619f30aeea6f4081711af8dc4481db748955ef288b66f89f0f4: Status 404 returned error can't find the container with id e8681336d6796619f30aeea6f4081711af8dc4481db748955ef288b66f89f0f4
Oct 30 05:36:21.205412 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:21.205391    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"]
Oct 30 05:36:21.441686 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:21.441644    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv" event={"ID":"3b61e548-7a4e-4c51-b45d-9f87de504c48","Type":"ContainerStarted","Data":"e8681336d6796619f30aeea6f4081711af8dc4481db748955ef288b66f89f0f4"}
Oct 30 05:36:22.451129 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:22.451092    5072 generic.go:356] "Generic (PLEG): container finished" podID="3b61e548-7a4e-4c51-b45d-9f87de504c48" containerID="65ecc861a64ad65f1f9de83cb9ef778d927489749ad1a0e10bc06d9a0252a041" exitCode=0
Oct 30 05:36:22.451673 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:22.451166    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv" event={"ID":"3b61e548-7a4e-4c51-b45d-9f87de504c48","Type":"ContainerDied","Data":"65ecc861a64ad65f1f9de83cb9ef778d927489749ad1a0e10bc06d9a0252a041"}
Oct 30 05:36:23.924315 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:23.923930    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:23.950343 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:23.950308    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-xks-ca\") pod \"3b61e548-7a4e-4c51-b45d-9f87de504c48\" (UID: \"3b61e548-7a4e-4c51-b45d-9f87de504c48\") "
Oct 30 05:36:23.950515 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:23.950445    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-openshift-ca\") pod \"3b61e548-7a4e-4c51-b45d-9f87de504c48\" (UID: \"3b61e548-7a4e-4c51-b45d-9f87de504c48\") "
Oct 30 05:36:23.950515 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:23.950469    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b61e548-7a4e-4c51-b45d-9f87de504c48-kube-api-access-kcrwz\" (UniqueName: \"kubernetes.io/projected/3b61e548-7a4e-4c51-b45d-9f87de504c48-kube-api-access-kcrwz\") pod \"3b61e548-7a4e-4c51-b45d-9f87de504c48\" (UID: \"3b61e548-7a4e-4c51-b45d-9f87de504c48\") "
Oct 30 05:36:23.950918 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:23.950888    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-openshift-ca" pod "3b61e548-7a4e-4c51-b45d-9f87de504c48" (UID: "3b61e548-7a4e-4c51-b45d-9f87de504c48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:36:23.952873 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:23.952849    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b61e548-7a4e-4c51-b45d-9f87de504c48-kube-api-access-kcrwz" pod "3b61e548-7a4e-4c51-b45d-9f87de504c48" (UID: "3b61e548-7a4e-4c51-b45d-9f87de504c48"). InnerVolumeSpecName "kube-api-access-kcrwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:23.953206 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:23.953174    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-xks-ca" pod "3b61e548-7a4e-4c51-b45d-9f87de504c48" (UID: "3b61e548-7a4e-4c51-b45d-9f87de504c48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:36:24.051764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:24.051724    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:24.051764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:24.051759    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcrwz\" (UniqueName: \"kubernetes.io/projected/3b61e548-7a4e-4c51-b45d-9f87de504c48-kube-api-access-kcrwz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:24.051764 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:24.051769    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b61e548-7a4e-4c51-b45d-9f87de504c48-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:24.494102 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:24.493955    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv" event={"ID":"3b61e548-7a4e-4c51-b45d-9f87de504c48","Type":"ContainerDied","Data":"e8681336d6796619f30aeea6f4081711af8dc4481db748955ef288b66f89f0f4"}
Oct 30 05:36:24.494102 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:24.494021    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8681336d6796619f30aeea6f4081711af8dc4481db748955ef288b66f89f0f4"
Oct 30 05:36:24.494380 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:24.494134    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08dbc435-2ccbv"
Oct 30 05:36:31.055354 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.055304    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"]
Oct 30 05:36:31.056044 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.055817    5072 cpu_manager.go:414] "containerMap: removing container" podUID="3b61e548-7a4e-4c51-b45d-9f87de504c48" containerName="revoke-keys"
Oct 30 05:36:31.056044 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.055830    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b61e548-7a4e-4c51-b45d-9f87de504c48" containerName="revoke-keys"
Oct 30 05:36:31.056044 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.055916    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b61e548-7a4e-4c51-b45d-9f87de504c48" containerName="revoke-keys"
Oct 30 05:36:31.059525 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.059499    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:31.078419 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.078387    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"]
Oct 30 05:36:31.087680 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.087650    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:36:31.087900 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.087684    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:36:31.087900 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.087714    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:36:31.087900 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.087815    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:36:31.121357 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.121319    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df35c8ed-drqcs\" (UID: \"14ffe21e-64b5-43d1-91d9-092d819fd727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:31.121541 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.121379    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ddv\" (UniqueName: \"kubernetes.io/projected/14ffe21e-64b5-43d1-91d9-092d819fd727-kube-api-access-c4ddv\") pod \"maas-api-revoke-sub-df35c8ed-drqcs\" (UID: \"14ffe21e-64b5-43d1-91d9-092d819fd727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:31.121541 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.121438    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df35c8ed-drqcs\" (UID: \"14ffe21e-64b5-43d1-91d9-092d819fd727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:31.222596 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.222563    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df35c8ed-drqcs\" (UID: \"14ffe21e-64b5-43d1-91d9-092d819fd727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:31.222779 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.222619    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ddv\" (UniqueName: \"kubernetes.io/projected/14ffe21e-64b5-43d1-91d9-092d819fd727-kube-api-access-c4ddv\") pod \"maas-api-revoke-sub-df35c8ed-drqcs\" (UID: \"14ffe21e-64b5-43d1-91d9-092d819fd727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:31.222779 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.222642    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df35c8ed-drqcs\" (UID: \"14ffe21e-64b5-43d1-91d9-092d819fd727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:31.223443 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.223377    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df35c8ed-drqcs\" (UID: \"14ffe21e-64b5-43d1-91d9-092d819fd727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:31.225266 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.225242    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df35c8ed-drqcs\" (UID: \"14ffe21e-64b5-43d1-91d9-092d819fd727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:31.246913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.246887    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ddv\" (UniqueName: \"kubernetes.io/projected/14ffe21e-64b5-43d1-91d9-092d819fd727-kube-api-access-c4ddv\") pod \"maas-api-revoke-sub-df35c8ed-drqcs\" (UID: \"14ffe21e-64b5-43d1-91d9-092d819fd727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:31.373445 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.373355    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:31.680871 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:31.680846    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"]
Oct 30 05:36:31.681663 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:36:31.681641    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ffe21e_64b5_43d1_91d9_092d819fd727.slice/crio-e0d7d13b756cd94711434b227d06b698c67b5ef1ad631f618aed6c709a62cff0 WatchSource:0}: Error finding container e0d7d13b756cd94711434b227d06b698c67b5ef1ad631f618aed6c709a62cff0: Status 404 returned error can't find the container with id e0d7d13b756cd94711434b227d06b698c67b5ef1ad631f618aed6c709a62cff0
Oct 30 05:36:32.541227 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:32.541186    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs" event={"ID":"14ffe21e-64b5-43d1-91d9-092d819fd727","Type":"ContainerStarted","Data":"4d3be9e55d364c2e3507ebbd826a0afd76088e8776f6d51ed1c689785f6a01bb"}
Oct 30 05:36:32.541751 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:32.541249    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs" event={"ID":"14ffe21e-64b5-43d1-91d9-092d819fd727","Type":"ContainerStarted","Data":"e0d7d13b756cd94711434b227d06b698c67b5ef1ad631f618aed6c709a62cff0"}
Oct 30 05:36:32.585950 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:32.585890    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs" podStartSLOduration=1.585872752 podStartE2EDuration="1.585872752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:32.583349357 +0000 UTC m=+5972.324772454" watchObservedRunningTime="2026-10-30 05:36:32.585872752 +0000 UTC m=+5972.327295851"
Oct 30 05:36:33.548965 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:33.548934    5072 generic.go:356] "Generic (PLEG): container finished" podID="14ffe21e-64b5-43d1-91d9-092d819fd727" containerID="4d3be9e55d364c2e3507ebbd826a0afd76088e8776f6d51ed1c689785f6a01bb" exitCode=0
Oct 30 05:36:33.549412 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:33.549019    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs" event={"ID":"14ffe21e-64b5-43d1-91d9-092d819fd727","Type":"ContainerDied","Data":"4d3be9e55d364c2e3507ebbd826a0afd76088e8776f6d51ed1c689785f6a01bb"}
Oct 30 05:36:34.874019 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:34.873679    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:34.979563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:34.979528    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-openshift-ca\") pod \"14ffe21e-64b5-43d1-91d9-092d819fd727\" (UID: \"14ffe21e-64b5-43d1-91d9-092d819fd727\") "
Oct 30 05:36:34.979740 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:34.979635    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-xks-ca\") pod \"14ffe21e-64b5-43d1-91d9-092d819fd727\" (UID: \"14ffe21e-64b5-43d1-91d9-092d819fd727\") "
Oct 30 05:36:34.979740 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:34.979666    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ffe21e-64b5-43d1-91d9-092d819fd727-kube-api-access-c4ddv\" (UniqueName: \"kubernetes.io/projected/14ffe21e-64b5-43d1-91d9-092d819fd727-kube-api-access-c4ddv\") pod \"14ffe21e-64b5-43d1-91d9-092d819fd727\" (UID: \"14ffe21e-64b5-43d1-91d9-092d819fd727\") "
Oct 30 05:36:34.979988 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:34.979930    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-openshift-ca" pod "14ffe21e-64b5-43d1-91d9-092d819fd727" (UID: "14ffe21e-64b5-43d1-91d9-092d819fd727"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:36:34.980154 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:34.980093    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:34.981973 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:34.981947    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-xks-ca" pod "14ffe21e-64b5-43d1-91d9-092d819fd727" (UID: "14ffe21e-64b5-43d1-91d9-092d819fd727"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:36:34.982107 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:34.981982    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ffe21e-64b5-43d1-91d9-092d819fd727-kube-api-access-c4ddv" pod "14ffe21e-64b5-43d1-91d9-092d819fd727" (UID: "14ffe21e-64b5-43d1-91d9-092d819fd727"). InnerVolumeSpecName "kube-api-access-c4ddv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:35.081225 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:35.081195    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ffe21e-64b5-43d1-91d9-092d819fd727-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:35.081225 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:35.081221    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4ddv\" (UniqueName: \"kubernetes.io/projected/14ffe21e-64b5-43d1-91d9-092d819fd727-kube-api-access-c4ddv\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:35.561878 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:35.561849    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs"
Oct 30 05:36:35.562070 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:35.561937    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df35c8ed-drqcs" event={"ID":"14ffe21e-64b5-43d1-91d9-092d819fd727","Type":"ContainerDied","Data":"e0d7d13b756cd94711434b227d06b698c67b5ef1ad631f618aed6c709a62cff0"}
Oct 30 05:36:35.562070 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:35.561986    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0d7d13b756cd94711434b227d06b698c67b5ef1ad631f618aed6c709a62cff0"
Oct 30 05:36:41.515558 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.515464    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"]
Oct 30 05:36:41.516306 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.516283    5072 cpu_manager.go:414] "containerMap: removing container" podUID="14ffe21e-64b5-43d1-91d9-092d819fd727" containerName="revoke-keys"
Oct 30 05:36:41.516372 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.516310    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ffe21e-64b5-43d1-91d9-092d819fd727" containerName="revoke-keys"
Oct 30 05:36:41.516496 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.516483    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ffe21e-64b5-43d1-91d9-092d819fd727" containerName="revoke-keys"
Oct 30 05:36:41.520241 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.520219    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:41.537635 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.537608    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"]
Oct 30 05:36:41.547437 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.547412    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:36:41.547616 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.547448    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:36:41.547616 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.547484    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:36:41.548554 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.548534    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:36:41.640213 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.640187    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9b26\" (UniqueName: \"kubernetes.io/projected/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-kube-api-access-s9b26\") pod \"maas-api-revoke-sub-40ebc570-7t879\" (UID: \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:41.640389 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.640255    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ebc570-7t879\" (UID: \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:41.640389 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.640354    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ebc570-7t879\" (UID: \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:41.741904 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.741858    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9b26\" (UniqueName: \"kubernetes.io/projected/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-kube-api-access-s9b26\") pod \"maas-api-revoke-sub-40ebc570-7t879\" (UID: \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:41.742131 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.741949    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ebc570-7t879\" (UID: \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:41.742131 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.741984    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ebc570-7t879\" (UID: \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:41.742753 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.742722    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ebc570-7t879\" (UID: \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:41.744592 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.744569    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ebc570-7t879\" (UID: \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:41.751409 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:36:41.751379    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:36:41.766800 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.766729    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9b26\" (UniqueName: \"kubernetes.io/projected/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-kube-api-access-s9b26\") pod \"maas-api-revoke-sub-40ebc570-7t879\" (UID: \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:41.833227 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:41.833201    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:42.135365 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:42.135332    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"]
Oct 30 05:36:42.136356 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:36:42.136332    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136f2f8e_45f5_434e_a719_6d9c4f0ea69b.slice/crio-2fa6cee602a1f8e84bcbeb7c11efa56777a55b8343d9e7d24b5be123177bf5be WatchSource:0}: Error finding container 2fa6cee602a1f8e84bcbeb7c11efa56777a55b8343d9e7d24b5be123177bf5be: Status 404 returned error can't find the container with id 2fa6cee602a1f8e84bcbeb7c11efa56777a55b8343d9e7d24b5be123177bf5be
Oct 30 05:36:42.603733 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:42.603700    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879" event={"ID":"136f2f8e-45f5-434e-a719-6d9c4f0ea69b","Type":"ContainerStarted","Data":"00cf17040dc23b313a5ead15d22e237f0cca2f8358e4030579ec3145d43b0842"}
Oct 30 05:36:42.604103 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:42.603756    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879" event={"ID":"136f2f8e-45f5-434e-a719-6d9c4f0ea69b","Type":"ContainerStarted","Data":"2fa6cee602a1f8e84bcbeb7c11efa56777a55b8343d9e7d24b5be123177bf5be"}
Oct 30 05:36:42.647813 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:42.647764    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879" podStartSLOduration=1.6477507120000001 podStartE2EDuration="1.647750712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:42.6455448 +0000 UTC m=+5982.386967896" watchObservedRunningTime="2026-10-30 05:36:42.647750712 +0000 UTC m=+5982.389173808"
Oct 30 05:36:43.612219 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:43.612180    5072 generic.go:356] "Generic (PLEG): container finished" podID="136f2f8e-45f5-434e-a719-6d9c4f0ea69b" containerID="00cf17040dc23b313a5ead15d22e237f0cca2f8358e4030579ec3145d43b0842" exitCode=0
Oct 30 05:36:43.612670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:43.612278    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879" event={"ID":"136f2f8e-45f5-434e-a719-6d9c4f0ea69b","Type":"ContainerDied","Data":"00cf17040dc23b313a5ead15d22e237f0cca2f8358e4030579ec3145d43b0842"}
Oct 30 05:36:44.986101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:44.966291    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:45.135925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:45.135144    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-openshift-ca\") pod \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\" (UID: \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\") "
Oct 30 05:36:45.135925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:45.135310    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-xks-ca\") pod \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\" (UID: \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\") "
Oct 30 05:36:45.135925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:45.135365    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-kube-api-access-s9b26\" (UniqueName: \"kubernetes.io/projected/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-kube-api-access-s9b26\") pod \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\" (UID: \"136f2f8e-45f5-434e-a719-6d9c4f0ea69b\") "
Oct 30 05:36:45.135925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:45.135580    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-openshift-ca" pod "136f2f8e-45f5-434e-a719-6d9c4f0ea69b" (UID: "136f2f8e-45f5-434e-a719-6d9c4f0ea69b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:36:45.135925 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:45.135693    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:45.138556 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:45.138436    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-xks-ca" pod "136f2f8e-45f5-434e-a719-6d9c4f0ea69b" (UID: "136f2f8e-45f5-434e-a719-6d9c4f0ea69b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:36:45.138766 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:45.138750    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-kube-api-access-s9b26" pod "136f2f8e-45f5-434e-a719-6d9c4f0ea69b" (UID: "136f2f8e-45f5-434e-a719-6d9c4f0ea69b"). InnerVolumeSpecName "kube-api-access-s9b26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:45.237139 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:45.237041    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:45.237139 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:45.237070    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9b26\" (UniqueName: \"kubernetes.io/projected/136f2f8e-45f5-434e-a719-6d9c4f0ea69b-kube-api-access-s9b26\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:45.634844 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:45.634759    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879"
Oct 30 05:36:45.635020 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:45.634753    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ebc570-7t879" event={"ID":"136f2f8e-45f5-434e-a719-6d9c4f0ea69b","Type":"ContainerDied","Data":"2fa6cee602a1f8e84bcbeb7c11efa56777a55b8343d9e7d24b5be123177bf5be"}
Oct 30 05:36:45.635020 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:45.634916    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fa6cee602a1f8e84bcbeb7c11efa56777a55b8343d9e7d24b5be123177bf5be"
Oct 30 05:36:51.888958 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.888925    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"]
Oct 30 05:36:51.889437 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.889424    5072 cpu_manager.go:414] "containerMap: removing container" podUID="136f2f8e-45f5-434e-a719-6d9c4f0ea69b" containerName="revoke-keys"
Oct 30 05:36:51.889485 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.889438    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136f2f8e-45f5-434e-a719-6d9c4f0ea69b" containerName="revoke-keys"
Oct 30 05:36:51.889556 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.889547    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="136f2f8e-45f5-434e-a719-6d9c4f0ea69b" containerName="revoke-keys"
Oct 30 05:36:51.893030 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.893006    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:36:51.912376 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.912348    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"]
Oct 30 05:36:51.921579 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.921504    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:36:51.921579 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.921520    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:36:51.921579 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.921538    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:36:51.921820 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.921798    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:36:51.994126 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.994067    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440fabe7-qgcbg\" (UID: \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:36:51.994288 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.994194    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpfjx\" (UniqueName: \"kubernetes.io/projected/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-kube-api-access-cpfjx\") pod \"maas-api-revoke-sub-440fabe7-qgcbg\" (UID: \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:36:51.994288 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:51.994268    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440fabe7-qgcbg\" (UID: \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:36:52.094716 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:52.094685    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpfjx\" (UniqueName: \"kubernetes.io/projected/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-kube-api-access-cpfjx\") pod \"maas-api-revoke-sub-440fabe7-qgcbg\" (UID: \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:36:52.094864 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:52.094733    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440fabe7-qgcbg\" (UID: \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:36:52.094864 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:52.094781    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440fabe7-qgcbg\" (UID: \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:36:52.095419 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:52.095388    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440fabe7-qgcbg\" (UID: \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:36:52.097297 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:52.097276    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440fabe7-qgcbg\" (UID: \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:36:52.119284 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:52.119255    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpfjx\" (UniqueName: \"kubernetes.io/projected/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-kube-api-access-cpfjx\") pod \"maas-api-revoke-sub-440fabe7-qgcbg\" (UID: \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:36:52.212190 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:52.212166    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:36:52.511384 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:36:52.511349    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ba74f6_f1cf_4bd1_bcfd_2f595f033025.slice/crio-6aa555f4ff8c4205778775ffae9c0f60c9122f471ab38f8051451084142c2e3e WatchSource:0}: Error finding container 6aa555f4ff8c4205778775ffae9c0f60c9122f471ab38f8051451084142c2e3e: Status 404 returned error can't find the container with id 6aa555f4ff8c4205778775ffae9c0f60c9122f471ab38f8051451084142c2e3e
Oct 30 05:36:52.511384 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:52.511367    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"]
Oct 30 05:36:52.674840 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:52.674813    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg" event={"ID":"21ba74f6-f1cf-4bd1-bcfd-2f595f033025","Type":"ContainerStarted","Data":"6aa555f4ff8c4205778775ffae9c0f60c9122f471ab38f8051451084142c2e3e"}
Oct 30 05:36:53.682262 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:53.682165    5072 generic.go:356] "Generic (PLEG): container finished" podID="21ba74f6-f1cf-4bd1-bcfd-2f595f033025" containerID="68790ba166e902970721ff97c7a0061ec84c2847d9ce1a0ac3f86bb71722088e" exitCode=0
Oct 30 05:36:53.682676 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:53.682251    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg" event={"ID":"21ba74f6-f1cf-4bd1-bcfd-2f595f033025","Type":"ContainerDied","Data":"68790ba166e902970721ff97c7a0061ec84c2847d9ce1a0ac3f86bb71722088e"}
Oct 30 05:36:55.098112 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.097318    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:36:55.145169 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.145142    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-xks-ca\") pod \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\" (UID: \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\") "
Oct 30 05:36:55.145511 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.145483    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-kube-api-access-cpfjx\" (UniqueName: \"kubernetes.io/projected/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-kube-api-access-cpfjx\") pod \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\" (UID: \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\") "
Oct 30 05:36:55.145621 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.145591    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-openshift-ca\") pod \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\" (UID: \"21ba74f6-f1cf-4bd1-bcfd-2f595f033025\") "
Oct 30 05:36:55.146350 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.146319    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-openshift-ca" pod "21ba74f6-f1cf-4bd1-bcfd-2f595f033025" (UID: "21ba74f6-f1cf-4bd1-bcfd-2f595f033025"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:36:55.147661 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.147621    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-xks-ca" pod "21ba74f6-f1cf-4bd1-bcfd-2f595f033025" (UID: "21ba74f6-f1cf-4bd1-bcfd-2f595f033025"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:36:55.148096 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.148052    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-kube-api-access-cpfjx" pod "21ba74f6-f1cf-4bd1-bcfd-2f595f033025" (UID: "21ba74f6-f1cf-4bd1-bcfd-2f595f033025"). InnerVolumeSpecName "kube-api-access-cpfjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:55.246664 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.246619    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:55.246664 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.246663    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpfjx\" (UniqueName: \"kubernetes.io/projected/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-kube-api-access-cpfjx\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:55.246856 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.246681    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ba74f6-f1cf-4bd1-bcfd-2f595f033025-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:36:55.717030 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.716995    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg" event={"ID":"21ba74f6-f1cf-4bd1-bcfd-2f595f033025","Type":"ContainerDied","Data":"6aa555f4ff8c4205778775ffae9c0f60c9122f471ab38f8051451084142c2e3e"}
Oct 30 05:36:55.717402 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.717385    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6aa555f4ff8c4205778775ffae9c0f60c9122f471ab38f8051451084142c2e3e"
Oct 30 05:36:55.717524 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:36:55.717308    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440fabe7-qgcbg"
Oct 30 05:37:01.353396 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:01.353365    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:37:01.370638 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:01.370613    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:37:02.252785 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.252750    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"]
Oct 30 05:37:02.253322 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.253306    5072 cpu_manager.go:414] "containerMap: removing container" podUID="21ba74f6-f1cf-4bd1-bcfd-2f595f033025" containerName="revoke-keys"
Oct 30 05:37:02.253386 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.253324    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ba74f6-f1cf-4bd1-bcfd-2f595f033025" containerName="revoke-keys"
Oct 30 05:37:02.253445 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.253434    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ba74f6-f1cf-4bd1-bcfd-2f595f033025" containerName="revoke-keys"
Oct 30 05:37:02.256656 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.256634    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:02.291543 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.291498    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"]
Oct 30 05:37:02.291719 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.291668    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:37:02.291777 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.291746    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:37:02.291817 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.291808    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:37:02.291903 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.291889    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:37:02.316030 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.315998    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmn7m\" (UniqueName: \"kubernetes.io/projected/6428ca6d-5a07-4e1a-a99b-428054b598f7-kube-api-access-cmn7m\") pod \"maas-api-revoke-sub-fa303d24-x7c2x\" (UID: \"6428ca6d-5a07-4e1a-a99b-428054b598f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:02.316428 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.316057    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa303d24-x7c2x\" (UID: \"6428ca6d-5a07-4e1a-a99b-428054b598f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:02.316428 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.316180    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa303d24-x7c2x\" (UID: \"6428ca6d-5a07-4e1a-a99b-428054b598f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:02.417024 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.416988    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa303d24-x7c2x\" (UID: \"6428ca6d-5a07-4e1a-a99b-428054b598f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:02.417510 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.417102    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmn7m\" (UniqueName: \"kubernetes.io/projected/6428ca6d-5a07-4e1a-a99b-428054b598f7-kube-api-access-cmn7m\") pod \"maas-api-revoke-sub-fa303d24-x7c2x\" (UID: \"6428ca6d-5a07-4e1a-a99b-428054b598f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:02.417510 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.417153    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa303d24-x7c2x\" (UID: \"6428ca6d-5a07-4e1a-a99b-428054b598f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:02.417947 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.417923    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa303d24-x7c2x\" (UID: \"6428ca6d-5a07-4e1a-a99b-428054b598f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:02.419701 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.419676    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa303d24-x7c2x\" (UID: \"6428ca6d-5a07-4e1a-a99b-428054b598f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:02.442475 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.442443    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmn7m\" (UniqueName: \"kubernetes.io/projected/6428ca6d-5a07-4e1a-a99b-428054b598f7-kube-api-access-cmn7m\") pod \"maas-api-revoke-sub-fa303d24-x7c2x\" (UID: \"6428ca6d-5a07-4e1a-a99b-428054b598f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:02.570101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.570014    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:02.871346 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:02.871318    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"]
Oct 30 05:37:03.764830 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:03.764788    5072 generic.go:356] "Generic (PLEG): container finished" podID="6428ca6d-5a07-4e1a-a99b-428054b598f7" containerID="e8597bf12bbda1eddd239bfca2da6c6b38aa95f437f3c77f62a5c3cbf5e05b37" exitCode=0
Oct 30 05:37:03.765388 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:03.764868    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x" event={"ID":"6428ca6d-5a07-4e1a-a99b-428054b598f7","Type":"ContainerDied","Data":"e8597bf12bbda1eddd239bfca2da6c6b38aa95f437f3c77f62a5c3cbf5e05b37"}
Oct 30 05:37:03.765388 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:03.764927    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x" event={"ID":"6428ca6d-5a07-4e1a-a99b-428054b598f7","Type":"ContainerStarted","Data":"46b230cd046ba0da87e95c0fb64ecf613394395e56788ca0b3c09ba228c86512"}
Oct 30 05:37:05.214105 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.210495    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:05.248213 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.248186    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-xks-ca\") pod \"6428ca6d-5a07-4e1a-a99b-428054b598f7\" (UID: \"6428ca6d-5a07-4e1a-a99b-428054b598f7\") "
Oct 30 05:37:05.248366 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.248248    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6428ca6d-5a07-4e1a-a99b-428054b598f7-kube-api-access-cmn7m\" (UniqueName: \"kubernetes.io/projected/6428ca6d-5a07-4e1a-a99b-428054b598f7-kube-api-access-cmn7m\") pod \"6428ca6d-5a07-4e1a-a99b-428054b598f7\" (UID: \"6428ca6d-5a07-4e1a-a99b-428054b598f7\") "
Oct 30 05:37:05.248366 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.248280    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-openshift-ca\") pod \"6428ca6d-5a07-4e1a-a99b-428054b598f7\" (UID: \"6428ca6d-5a07-4e1a-a99b-428054b598f7\") "
Oct 30 05:37:05.248741 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.248718    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-openshift-ca" pod "6428ca6d-5a07-4e1a-a99b-428054b598f7" (UID: "6428ca6d-5a07-4e1a-a99b-428054b598f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:37:05.250578 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.250556    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-xks-ca" pod "6428ca6d-5a07-4e1a-a99b-428054b598f7" (UID: "6428ca6d-5a07-4e1a-a99b-428054b598f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:37:05.250669 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.250584    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6428ca6d-5a07-4e1a-a99b-428054b598f7-kube-api-access-cmn7m" pod "6428ca6d-5a07-4e1a-a99b-428054b598f7" (UID: "6428ca6d-5a07-4e1a-a99b-428054b598f7"). InnerVolumeSpecName "kube-api-access-cmn7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:05.349819 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.349788    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:05.349819 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.349817    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmn7m\" (UniqueName: \"kubernetes.io/projected/6428ca6d-5a07-4e1a-a99b-428054b598f7-kube-api-access-cmn7m\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:05.350011 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.349828    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6428ca6d-5a07-4e1a-a99b-428054b598f7-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:05.827250 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.827203    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x" event={"ID":"6428ca6d-5a07-4e1a-a99b-428054b598f7","Type":"ContainerDied","Data":"46b230cd046ba0da87e95c0fb64ecf613394395e56788ca0b3c09ba228c86512"}
Oct 30 05:37:05.827467 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.827271    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46b230cd046ba0da87e95c0fb64ecf613394395e56788ca0b3c09ba228c86512"
Oct 30 05:37:05.827467 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:05.827346    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa303d24-x7c2x"
Oct 30 05:37:11.751521 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:37:11.751478    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:37:12.613261 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.613228    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"]
Oct 30 05:37:12.613720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.613707    5072 cpu_manager.go:414] "containerMap: removing container" podUID="6428ca6d-5a07-4e1a-a99b-428054b598f7" containerName="revoke-keys"
Oct 30 05:37:12.613770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.613721    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6428ca6d-5a07-4e1a-a99b-428054b598f7" containerName="revoke-keys"
Oct 30 05:37:12.613808 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.613793    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6428ca6d-5a07-4e1a-a99b-428054b598f7" containerName="revoke-keys"
Oct 30 05:37:12.617313 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.617294    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:12.635465 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.635441    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"]
Oct 30 05:37:12.644720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.644685    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:37:12.644720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.644713    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:37:12.644917 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.644779    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:37:12.645641 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.645622    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:37:12.728147 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.728117    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d84d7abe-btcl4\" (UID: \"f54234dd-cceb-41e9-b7f3-504dc067885c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:12.728300 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.728170    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d84d7abe-btcl4\" (UID: \"f54234dd-cceb-41e9-b7f3-504dc067885c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:12.728300 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.728281    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt8vm\" (UniqueName: \"kubernetes.io/projected/f54234dd-cceb-41e9-b7f3-504dc067885c-kube-api-access-vt8vm\") pod \"maas-api-revoke-sub-d84d7abe-btcl4\" (UID: \"f54234dd-cceb-41e9-b7f3-504dc067885c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:12.829372 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.829342    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d84d7abe-btcl4\" (UID: \"f54234dd-cceb-41e9-b7f3-504dc067885c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:12.829854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.829393    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d84d7abe-btcl4\" (UID: \"f54234dd-cceb-41e9-b7f3-504dc067885c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:12.829854 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.829441    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt8vm\" (UniqueName: \"kubernetes.io/projected/f54234dd-cceb-41e9-b7f3-504dc067885c-kube-api-access-vt8vm\") pod \"maas-api-revoke-sub-d84d7abe-btcl4\" (UID: \"f54234dd-cceb-41e9-b7f3-504dc067885c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:12.830186 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.830165    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d84d7abe-btcl4\" (UID: \"f54234dd-cceb-41e9-b7f3-504dc067885c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:12.831996 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.831959    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d84d7abe-btcl4\" (UID: \"f54234dd-cceb-41e9-b7f3-504dc067885c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:12.854328 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.854299    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt8vm\" (UniqueName: \"kubernetes.io/projected/f54234dd-cceb-41e9-b7f3-504dc067885c-kube-api-access-vt8vm\") pod \"maas-api-revoke-sub-d84d7abe-btcl4\" (UID: \"f54234dd-cceb-41e9-b7f3-504dc067885c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:12.930703 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:12.930642    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:13.232788 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:37:13.232749    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54234dd_cceb_41e9_b7f3_504dc067885c.slice/crio-1db6f6293093cdb8b2268becb15aff45a9df513e3179ea91b9000eb9eb7f033b WatchSource:0}: Error finding container 1db6f6293093cdb8b2268becb15aff45a9df513e3179ea91b9000eb9eb7f033b: Status 404 returned error can't find the container with id 1db6f6293093cdb8b2268becb15aff45a9df513e3179ea91b9000eb9eb7f033b
Oct 30 05:37:13.233010 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:13.232988    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"]
Oct 30 05:37:13.872356 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:13.872279    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4" event={"ID":"f54234dd-cceb-41e9-b7f3-504dc067885c","Type":"ContainerStarted","Data":"53427d09a62a6276c8dd12f5a6fdcbd733721b38a40a4a79214f6798ec3c6459"}
Oct 30 05:37:13.872356 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:13.872336    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4" event={"ID":"f54234dd-cceb-41e9-b7f3-504dc067885c","Type":"ContainerStarted","Data":"1db6f6293093cdb8b2268becb15aff45a9df513e3179ea91b9000eb9eb7f033b"}
Oct 30 05:37:13.914391 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:13.914346    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4" podStartSLOduration=1.91433314 podStartE2EDuration="1.91433314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:13.913053203 +0000 UTC m=+6013.654476301" watchObservedRunningTime="2026-10-30 05:37:13.91433314 +0000 UTC m=+6013.655756237"
Oct 30 05:37:14.882397 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:14.882348    5072 generic.go:356] "Generic (PLEG): container finished" podID="f54234dd-cceb-41e9-b7f3-504dc067885c" containerID="53427d09a62a6276c8dd12f5a6fdcbd733721b38a40a4a79214f6798ec3c6459" exitCode=0
Oct 30 05:37:14.882874 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:14.882473    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4" event={"ID":"f54234dd-cceb-41e9-b7f3-504dc067885c","Type":"ContainerDied","Data":"53427d09a62a6276c8dd12f5a6fdcbd733721b38a40a4a79214f6798ec3c6459"}
Oct 30 05:37:16.327893 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.327541    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:16.429428 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.429385    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-xks-ca\") pod \"f54234dd-cceb-41e9-b7f3-504dc067885c\" (UID: \"f54234dd-cceb-41e9-b7f3-504dc067885c\") "
Oct 30 05:37:16.429817 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.429792    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-openshift-ca\") pod \"f54234dd-cceb-41e9-b7f3-504dc067885c\" (UID: \"f54234dd-cceb-41e9-b7f3-504dc067885c\") "
Oct 30 05:37:16.429995 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.429977    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54234dd-cceb-41e9-b7f3-504dc067885c-kube-api-access-vt8vm\" (UniqueName: \"kubernetes.io/projected/f54234dd-cceb-41e9-b7f3-504dc067885c-kube-api-access-vt8vm\") pod \"f54234dd-cceb-41e9-b7f3-504dc067885c\" (UID: \"f54234dd-cceb-41e9-b7f3-504dc067885c\") "
Oct 30 05:37:16.431630 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.431605    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-openshift-ca" pod "f54234dd-cceb-41e9-b7f3-504dc067885c" (UID: "f54234dd-cceb-41e9-b7f3-504dc067885c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:37:16.439487 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.439448    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54234dd-cceb-41e9-b7f3-504dc067885c-kube-api-access-vt8vm" pod "f54234dd-cceb-41e9-b7f3-504dc067885c" (UID: "f54234dd-cceb-41e9-b7f3-504dc067885c"). InnerVolumeSpecName "kube-api-access-vt8vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:16.441642 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.441607    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-xks-ca" pod "f54234dd-cceb-41e9-b7f3-504dc067885c" (UID: "f54234dd-cceb-41e9-b7f3-504dc067885c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:37:16.532000 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.531902    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt8vm\" (UniqueName: \"kubernetes.io/projected/f54234dd-cceb-41e9-b7f3-504dc067885c-kube-api-access-vt8vm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:16.532000 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.531933    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:16.532000 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.531944    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54234dd-cceb-41e9-b7f3-504dc067885c-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:16.916103 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.916004    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4" event={"ID":"f54234dd-cceb-41e9-b7f3-504dc067885c","Type":"ContainerDied","Data":"1db6f6293093cdb8b2268becb15aff45a9df513e3179ea91b9000eb9eb7f033b"}
Oct 30 05:37:16.916103 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.916060    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db6f6293093cdb8b2268becb15aff45a9df513e3179ea91b9000eb9eb7f033b"
Oct 30 05:37:16.916103 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:16.916016    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d84d7abe-btcl4"
Oct 30 05:37:22.977946 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:22.977898    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"]
Oct 30 05:37:22.978571 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:22.978444    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f54234dd-cceb-41e9-b7f3-504dc067885c" containerName="revoke-keys"
Oct 30 05:37:22.978571 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:22.978460    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54234dd-cceb-41e9-b7f3-504dc067885c" containerName="revoke-keys"
Oct 30 05:37:22.978752 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:22.978582    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54234dd-cceb-41e9-b7f3-504dc067885c" containerName="revoke-keys"
Oct 30 05:37:22.981807 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:22.981784    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:22.999623 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:22.999597    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"]
Oct 30 05:37:23.002272 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.002245    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdf757ae-nj74c\" (UID: \"321853a1-5de7-4151-9c0c-abe5597c64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:23.002393 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.002299    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4264k\" (UniqueName: \"kubernetes.io/projected/321853a1-5de7-4151-9c0c-abe5597c64a7-kube-api-access-4264k\") pod \"maas-api-revoke-sub-fdf757ae-nj74c\" (UID: \"321853a1-5de7-4151-9c0c-abe5597c64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:23.002582 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.002558    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdf757ae-nj74c\" (UID: \"321853a1-5de7-4151-9c0c-abe5597c64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:23.009297 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.009271    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:37:23.009470 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.009294    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:37:23.009470 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.009309    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:37:23.009470 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.009331    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:37:23.103520 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.103481    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdf757ae-nj74c\" (UID: \"321853a1-5de7-4151-9c0c-abe5597c64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:23.103703 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.103529    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdf757ae-nj74c\" (UID: \"321853a1-5de7-4151-9c0c-abe5597c64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:23.103703 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.103551    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4264k\" (UniqueName: \"kubernetes.io/projected/321853a1-5de7-4151-9c0c-abe5597c64a7-kube-api-access-4264k\") pod \"maas-api-revoke-sub-fdf757ae-nj74c\" (UID: \"321853a1-5de7-4151-9c0c-abe5597c64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:23.104280 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.104253    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdf757ae-nj74c\" (UID: \"321853a1-5de7-4151-9c0c-abe5597c64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:23.106137 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.106116    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdf757ae-nj74c\" (UID: \"321853a1-5de7-4151-9c0c-abe5597c64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:23.128031 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.128005    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4264k\" (UniqueName: \"kubernetes.io/projected/321853a1-5de7-4151-9c0c-abe5597c64a7-kube-api-access-4264k\") pod \"maas-api-revoke-sub-fdf757ae-nj74c\" (UID: \"321853a1-5de7-4151-9c0c-abe5597c64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:23.294750 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.294682    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:23.597654 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:37:23.597624    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321853a1_5de7_4151_9c0c_abe5597c64a7.slice/crio-56917c74f0aafe468f369da20fc226c031d44605f4f59d544bab8f0f09eed045 WatchSource:0}: Error finding container 56917c74f0aafe468f369da20fc226c031d44605f4f59d544bab8f0f09eed045: Status 404 returned error can't find the container with id 56917c74f0aafe468f369da20fc226c031d44605f4f59d544bab8f0f09eed045
Oct 30 05:37:23.598217 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.598194    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"]
Oct 30 05:37:23.957037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.956999    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c" event={"ID":"321853a1-5de7-4151-9c0c-abe5597c64a7","Type":"ContainerStarted","Data":"b9c60c085d035d72d52357d9ce35f1eef74ee2f950918c45acc1a48bf3ad53b5"}
Oct 30 05:37:23.957184 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:23.957065    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c" event={"ID":"321853a1-5de7-4151-9c0c-abe5597c64a7","Type":"ContainerStarted","Data":"56917c74f0aafe468f369da20fc226c031d44605f4f59d544bab8f0f09eed045"}
Oct 30 05:37:24.001799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:24.001752    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c" podStartSLOduration=2.001739564 podStartE2EDuration="2.001739564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:23.999095544 +0000 UTC m=+6023.740518686" watchObservedRunningTime="2026-10-30 05:37:24.001739564 +0000 UTC m=+6023.743162661"
Oct 30 05:37:24.965915 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:24.965873    5072 generic.go:356] "Generic (PLEG): container finished" podID="321853a1-5de7-4151-9c0c-abe5597c64a7" containerID="b9c60c085d035d72d52357d9ce35f1eef74ee2f950918c45acc1a48bf3ad53b5" exitCode=0
Oct 30 05:37:24.966141 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:24.965962    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c" event={"ID":"321853a1-5de7-4151-9c0c-abe5597c64a7","Type":"ContainerDied","Data":"b9c60c085d035d72d52357d9ce35f1eef74ee2f950918c45acc1a48bf3ad53b5"}
Oct 30 05:37:26.354954 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:26.354928    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:26.445061 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:26.445031    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-xks-ca\") pod \"321853a1-5de7-4151-9c0c-abe5597c64a7\" (UID: \"321853a1-5de7-4151-9c0c-abe5597c64a7\") "
Oct 30 05:37:26.445242 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:26.445197    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321853a1-5de7-4151-9c0c-abe5597c64a7-kube-api-access-4264k\" (UniqueName: \"kubernetes.io/projected/321853a1-5de7-4151-9c0c-abe5597c64a7-kube-api-access-4264k\") pod \"321853a1-5de7-4151-9c0c-abe5597c64a7\" (UID: \"321853a1-5de7-4151-9c0c-abe5597c64a7\") "
Oct 30 05:37:26.445295 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:26.445251    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-openshift-ca\") pod \"321853a1-5de7-4151-9c0c-abe5597c64a7\" (UID: \"321853a1-5de7-4151-9c0c-abe5597c64a7\") "
Oct 30 05:37:26.445630 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:26.445600    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-openshift-ca" pod "321853a1-5de7-4151-9c0c-abe5597c64a7" (UID: "321853a1-5de7-4151-9c0c-abe5597c64a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:37:26.447365 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:26.447341    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-xks-ca" pod "321853a1-5de7-4151-9c0c-abe5597c64a7" (UID: "321853a1-5de7-4151-9c0c-abe5597c64a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:37:26.447491 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:26.447429    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321853a1-5de7-4151-9c0c-abe5597c64a7-kube-api-access-4264k" pod "321853a1-5de7-4151-9c0c-abe5597c64a7" (UID: "321853a1-5de7-4151-9c0c-abe5597c64a7"). InnerVolumeSpecName "kube-api-access-4264k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:26.546204 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:26.546172    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4264k\" (UniqueName: \"kubernetes.io/projected/321853a1-5de7-4151-9c0c-abe5597c64a7-kube-api-access-4264k\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:26.546204 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:26.546201    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:26.546204 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:26.546213    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/321853a1-5de7-4151-9c0c-abe5597c64a7-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:27.003092 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:27.003046    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c"
Oct 30 05:37:27.003250 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:27.003088    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdf757ae-nj74c" event={"ID":"321853a1-5de7-4151-9c0c-abe5597c64a7","Type":"ContainerDied","Data":"56917c74f0aafe468f369da20fc226c031d44605f4f59d544bab8f0f09eed045"}
Oct 30 05:37:27.003250 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:27.003135    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56917c74f0aafe468f369da20fc226c031d44605f4f59d544bab8f0f09eed045"
Oct 30 05:37:33.339435 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.339398    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"]
Oct 30 05:37:33.340158 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.340136    5072 cpu_manager.go:414] "containerMap: removing container" podUID="321853a1-5de7-4151-9c0c-abe5597c64a7" containerName="revoke-keys"
Oct 30 05:37:33.340158 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.340158    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321853a1-5de7-4151-9c0c-abe5597c64a7" containerName="revoke-keys"
Oct 30 05:37:33.340344 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.340282    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="321853a1-5de7-4151-9c0c-abe5597c64a7" containerName="revoke-keys"
Oct 30 05:37:33.344120 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.344101    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:33.376122 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.372916    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:37:33.376122 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.372936    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:37:33.376122 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.373067    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:37:33.376122 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.373147    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:37:33.377868 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.377843    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"]
Oct 30 05:37:33.413430 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.413400    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd2c403-ntkcx\" (UID: \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:33.413670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.413447    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd2c403-ntkcx\" (UID: \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:33.413670 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.413600    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdz5z\" (UniqueName: \"kubernetes.io/projected/f34d23d0-601e-46df-bbb6-96126cfbc9c6-kube-api-access-gdz5z\") pod \"maas-api-revoke-sub-dbd2c403-ntkcx\" (UID: \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:33.514913 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.514874    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdz5z\" (UniqueName: \"kubernetes.io/projected/f34d23d0-601e-46df-bbb6-96126cfbc9c6-kube-api-access-gdz5z\") pod \"maas-api-revoke-sub-dbd2c403-ntkcx\" (UID: \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:33.515125 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.514956    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd2c403-ntkcx\" (UID: \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:33.515125 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.515003    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd2c403-ntkcx\" (UID: \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:33.515652 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.515628    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd2c403-ntkcx\" (UID: \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:33.517635 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.517612    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd2c403-ntkcx\" (UID: \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:33.541603 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.541576    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdz5z\" (UniqueName: \"kubernetes.io/projected/f34d23d0-601e-46df-bbb6-96126cfbc9c6-kube-api-access-gdz5z\") pod \"maas-api-revoke-sub-dbd2c403-ntkcx\" (UID: \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:33.658493 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.658427    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:33.965928 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:33.965899    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"]
Oct 30 05:37:33.966330 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:37:33.966305    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34d23d0_601e_46df_bbb6_96126cfbc9c6.slice/crio-0396d22710fb2a48bce57d13e02c84aefebbf8277070efbafb7305e75c953591 WatchSource:0}: Error finding container 0396d22710fb2a48bce57d13e02c84aefebbf8277070efbafb7305e75c953591: Status 404 returned error can't find the container with id 0396d22710fb2a48bce57d13e02c84aefebbf8277070efbafb7305e75c953591
Oct 30 05:37:34.046999 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:34.046969    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx" event={"ID":"f34d23d0-601e-46df-bbb6-96126cfbc9c6","Type":"ContainerStarted","Data":"0396d22710fb2a48bce57d13e02c84aefebbf8277070efbafb7305e75c953591"}
Oct 30 05:37:35.058278 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:35.058241    5072 generic.go:356] "Generic (PLEG): container finished" podID="f34d23d0-601e-46df-bbb6-96126cfbc9c6" containerID="ca3de4470b11fd25225f4a93090d84862756977f34911de24148de6690a31000" exitCode=0
Oct 30 05:37:35.058695 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:35.058299    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx" event={"ID":"f34d23d0-601e-46df-bbb6-96126cfbc9c6","Type":"ContainerDied","Data":"ca3de4470b11fd25225f4a93090d84862756977f34911de24148de6690a31000"}
Oct 30 05:37:36.540699 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:36.540673    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:36.651361 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:36.651325    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-openshift-ca\") pod \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\" (UID: \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\") "
Oct 30 05:37:36.651361 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:36.651360    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34d23d0-601e-46df-bbb6-96126cfbc9c6-kube-api-access-gdz5z\" (UniqueName: \"kubernetes.io/projected/f34d23d0-601e-46df-bbb6-96126cfbc9c6-kube-api-access-gdz5z\") pod \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\" (UID: \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\") "
Oct 30 05:37:36.651597 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:36.651461    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-xks-ca\") pod \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\" (UID: \"f34d23d0-601e-46df-bbb6-96126cfbc9c6\") "
Oct 30 05:37:36.651698 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:36.651673    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-openshift-ca" pod "f34d23d0-601e-46df-bbb6-96126cfbc9c6" (UID: "f34d23d0-601e-46df-bbb6-96126cfbc9c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:37:36.653631 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:36.653599    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-xks-ca" pod "f34d23d0-601e-46df-bbb6-96126cfbc9c6" (UID: "f34d23d0-601e-46df-bbb6-96126cfbc9c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:37:36.653754 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:36.653700    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34d23d0-601e-46df-bbb6-96126cfbc9c6-kube-api-access-gdz5z" pod "f34d23d0-601e-46df-bbb6-96126cfbc9c6" (UID: "f34d23d0-601e-46df-bbb6-96126cfbc9c6"). InnerVolumeSpecName "kube-api-access-gdz5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:36.752940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:36.752871    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:36.752940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:36.752897    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34d23d0-601e-46df-bbb6-96126cfbc9c6-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:36.752940 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:36.752906    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdz5z\" (UniqueName: \"kubernetes.io/projected/f34d23d0-601e-46df-bbb6-96126cfbc9c6-kube-api-access-gdz5z\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:37.111153 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:37.107910    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx" event={"ID":"f34d23d0-601e-46df-bbb6-96126cfbc9c6","Type":"ContainerDied","Data":"0396d22710fb2a48bce57d13e02c84aefebbf8277070efbafb7305e75c953591"}
Oct 30 05:37:37.111153 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:37.107973    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0396d22710fb2a48bce57d13e02c84aefebbf8277070efbafb7305e75c953591"
Oct 30 05:37:37.111153 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:37.108044    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd2c403-ntkcx"
Oct 30 05:37:41.750419 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:37:41.750369    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:37:43.709720 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.709689    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"]
Oct 30 05:37:43.710217 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.710202    5072 cpu_manager.go:414] "containerMap: removing container" podUID="f34d23d0-601e-46df-bbb6-96126cfbc9c6" containerName="revoke-keys"
Oct 30 05:37:43.710278 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.710218    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34d23d0-601e-46df-bbb6-96126cfbc9c6" containerName="revoke-keys"
Oct 30 05:37:43.710316 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.710292    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34d23d0-601e-46df-bbb6-96126cfbc9c6" containerName="revoke-keys"
Oct 30 05:37:43.713824 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.713805    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:43.730351 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.730315    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:37:43.733390 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.733364    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"]
Oct 30 05:37:43.741844 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.741821    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:37:43.741946 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.741854    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:37:43.741946 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.741934    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:37:43.829695 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.829662    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgzh4\" (UniqueName: \"kubernetes.io/projected/62ef5729-9da4-4869-a370-e58c003c9916-kube-api-access-mgzh4\") pod \"maas-api-revoke-sub-fc584f05-75t2b\" (UID: \"62ef5729-9da4-4869-a370-e58c003c9916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:43.829808 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.829721    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc584f05-75t2b\" (UID: \"62ef5729-9da4-4869-a370-e58c003c9916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:43.829866 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.829801    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc584f05-75t2b\" (UID: \"62ef5729-9da4-4869-a370-e58c003c9916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:43.931043 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.931015    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgzh4\" (UniqueName: \"kubernetes.io/projected/62ef5729-9da4-4869-a370-e58c003c9916-kube-api-access-mgzh4\") pod \"maas-api-revoke-sub-fc584f05-75t2b\" (UID: \"62ef5729-9da4-4869-a370-e58c003c9916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:43.931180 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.931051    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc584f05-75t2b\" (UID: \"62ef5729-9da4-4869-a370-e58c003c9916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:43.931180 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.931070    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc584f05-75t2b\" (UID: \"62ef5729-9da4-4869-a370-e58c003c9916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:43.931755 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.931736    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc584f05-75t2b\" (UID: \"62ef5729-9da4-4869-a370-e58c003c9916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:43.933601 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.933584    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc584f05-75t2b\" (UID: \"62ef5729-9da4-4869-a370-e58c003c9916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:43.955399 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:43.955372    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgzh4\" (UniqueName: \"kubernetes.io/projected/62ef5729-9da4-4869-a370-e58c003c9916-kube-api-access-mgzh4\") pod \"maas-api-revoke-sub-fc584f05-75t2b\" (UID: \"62ef5729-9da4-4869-a370-e58c003c9916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:44.027111 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:44.027014    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:44.338013 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:44.337975    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"]
Oct 30 05:37:44.338562 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:37:44.338535    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ef5729_9da4_4869_a370_e58c003c9916.slice/crio-3bcd07c3704350d5c16dd6c76010f77e8f9b2c03c037d58ab086b41ad68e768a WatchSource:0}: Error finding container 3bcd07c3704350d5c16dd6c76010f77e8f9b2c03c037d58ab086b41ad68e768a: Status 404 returned error can't find the container with id 3bcd07c3704350d5c16dd6c76010f77e8f9b2c03c037d58ab086b41ad68e768a
Oct 30 05:37:45.175410 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:45.169535    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b" event={"ID":"62ef5729-9da4-4869-a370-e58c003c9916","Type":"ContainerStarted","Data":"e5f87969ef7aeae75b5cc12b4f0c54dd4f48a43d8a67e72f8ebc1011b7b0fc82"}
Oct 30 05:37:45.175410 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:45.169609    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b" event={"ID":"62ef5729-9da4-4869-a370-e58c003c9916","Type":"ContainerStarted","Data":"3bcd07c3704350d5c16dd6c76010f77e8f9b2c03c037d58ab086b41ad68e768a"}
Oct 30 05:37:45.323102 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:45.320470    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b" podStartSLOduration=2.320449836 podStartE2EDuration="2.320449836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:45.311557715 +0000 UTC m=+6045.052980813" watchObservedRunningTime="2026-10-30 05:37:45.320449836 +0000 UTC m=+6045.061872934"
Oct 30 05:37:46.229101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:46.228859    5072 generic.go:356] "Generic (PLEG): container finished" podID="62ef5729-9da4-4869-a370-e58c003c9916" containerID="e5f87969ef7aeae75b5cc12b4f0c54dd4f48a43d8a67e72f8ebc1011b7b0fc82" exitCode=0
Oct 30 05:37:46.229101 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:46.229030    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b" event={"ID":"62ef5729-9da4-4869-a370-e58c003c9916","Type":"ContainerDied","Data":"e5f87969ef7aeae75b5cc12b4f0c54dd4f48a43d8a67e72f8ebc1011b7b0fc82"}
Oct 30 05:37:47.368824 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:47.368801    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:47.476318 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:47.476282    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-xks-ca\") pod \"62ef5729-9da4-4869-a370-e58c003c9916\" (UID: \"62ef5729-9da4-4869-a370-e58c003c9916\") "
Oct 30 05:37:47.476502 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:47.476331    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ef5729-9da4-4869-a370-e58c003c9916-kube-api-access-mgzh4\" (UniqueName: \"kubernetes.io/projected/62ef5729-9da4-4869-a370-e58c003c9916-kube-api-access-mgzh4\") pod \"62ef5729-9da4-4869-a370-e58c003c9916\" (UID: \"62ef5729-9da4-4869-a370-e58c003c9916\") "
Oct 30 05:37:47.476502 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:47.476488    5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-openshift-ca\") pod \"62ef5729-9da4-4869-a370-e58c003c9916\" (UID: \"62ef5729-9da4-4869-a370-e58c003c9916\") "
Oct 30 05:37:47.476847 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:47.476826    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-openshift-ca" pod "62ef5729-9da4-4869-a370-e58c003c9916" (UID: "62ef5729-9da4-4869-a370-e58c003c9916"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:37:47.478562 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:47.478535    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-xks-ca" pod "62ef5729-9da4-4869-a370-e58c003c9916" (UID: "62ef5729-9da4-4869-a370-e58c003c9916"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:37:47.478671 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:47.478581    5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ef5729-9da4-4869-a370-e58c003c9916-kube-api-access-mgzh4" pod "62ef5729-9da4-4869-a370-e58c003c9916" (UID: "62ef5729-9da4-4869-a370-e58c003c9916"). InnerVolumeSpecName "kube-api-access-mgzh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:47.577830 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:47.577804    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-xks-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:47.577830 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:47.577826    5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgzh4\" (UniqueName: \"kubernetes.io/projected/62ef5729-9da4-4869-a370-e58c003c9916-kube-api-access-mgzh4\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:47.577987 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:47.577836    5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ef5729-9da4-4869-a370-e58c003c9916-maas-api-openshift-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Oct 30 05:37:48.241735 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:48.241702    5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b"
Oct 30 05:37:48.241932 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:48.241746    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc584f05-75t2b" event={"ID":"62ef5729-9da4-4869-a370-e58c003c9916","Type":"ContainerDied","Data":"3bcd07c3704350d5c16dd6c76010f77e8f9b2c03c037d58ab086b41ad68e768a"}
Oct 30 05:37:48.241932 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:48.241803    5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bcd07c3704350d5c16dd6c76010f77e8f9b2c03c037d58ab086b41ad68e768a"
Oct 30 05:37:57.001261 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:57.001206    5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-txlk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:37:57.001975 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:37:57.001928    5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-txlk4" podUID="b748c4e2-2e95-43e6-8dc4-9bf3958a24d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:38:11.751254 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:38:11.751173    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:38:16.478754 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:16.478721    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6fxf_a80b6be8-ec4e-40c6-8889-256a744263b3/manager/0.log"
Oct 30 05:38:16.682770 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:16.682741    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qgl9w_a3f48faf-a0f6-4a0a-ac98-963183990dab/manager/0.log"
Oct 30 05:38:16.896623 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:16.896537    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-k6qmc_6671eae0-dd1d-41f5-b479-2c8e36e14650/manager/0.log"
Oct 30 05:38:17.445619 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:17.445594    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g9hm4_812448a2-1e32-4f19-86f0-8cf17f32edd1/server/0.log"
Oct 30 05:38:17.884996 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:17.884909    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-mb5bq_c7bec9d6-a8f4-4051-867a-b28db0efb1ed/proxy/0.log"
Oct 30 05:38:19.284698 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:19.284670    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-08dbc435-2ccbv_3b61e548-7a4e-4c51-b45d-9f87de504c48/revoke-keys/0.log"
Oct 30 05:38:19.477759 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:19.477729    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-23468440-kxzj4_b6e86583-d955-43ef-a522-89aac23f0526/revoke-keys/0.log"
Oct 30 05:38:19.663906 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:19.663836    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3ff93de4-pf8dj_da4e97a8-c2e7-4922-9b04-42ef3182518b/revoke-keys/0.log"
Oct 30 05:38:19.853482 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:19.853454    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-40ebc570-7t879_136f2f8e-45f5-434e-a719-6d9c4f0ea69b/revoke-keys/0.log"
Oct 30 05:38:20.044398 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:20.044357    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-440fabe7-qgcbg_21ba74f6-f1cf-4bd1-bcfd-2f595f033025/revoke-keys/0.log"
Oct 30 05:38:20.222696 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:20.222664    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-470c2cbb-5wznj_c2a0f085-bff2-442a-8af2-6bcf36d3f47f/revoke-keys/0.log"
Oct 30 05:38:20.409612 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:20.409535    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4ef228b2-dvchg_f2ce1164-ed6b-4751-98c6-b2e6641bf27d/revoke-keys/0.log"
Oct 30 05:38:20.599053 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:20.599010    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4fdc02eb-4gncp_6f44dc83-903b-403e-ad00-049dac8b7636/revoke-keys/0.log"
Oct 30 05:38:20.795381 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:20.795351    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-85b66617-q5cb2_2b6cc501-417e-4a72-9331-b8ea9354f2b2/revoke-keys/0.log"
Oct 30 05:38:20.978043 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:20.978016    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8d4ed2e9-wszd6_115b8606-cbec-4642-9dc9-61abefa440af/revoke-keys/0.log"
Oct 30 05:38:21.168015 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:21.167939    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d84d7abe-btcl4_f54234dd-cceb-41e9-b7f3-504dc067885c/revoke-keys/0.log"
Oct 30 05:38:21.355608 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:21.355578    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dbd2c403-ntkcx_f34d23d0-601e-46df-bbb6-96126cfbc9c6/revoke-keys/0.log"
Oct 30 05:38:21.536981 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:21.536948    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-df35c8ed-drqcs_14ffe21e-64b5-43d1-91d9-092d819fd727/revoke-keys/0.log"
Oct 30 05:38:21.721999 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:21.721971    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e42eb7ab-pv7pd_56f62958-7c41-45f5-83b2-83abfebe5bca/revoke-keys/0.log"
Oct 30 05:38:21.902893 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:21.902828    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e691efff-dz2ml_a6a28035-dce5-45fe-8a79-694a5c2ea708/revoke-keys/0.log"
Oct 30 05:38:22.078867 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:22.078840    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f5d6c54a-xtjnr_a111c0cb-6dc1-40f9-aa51-c0a861c305aa/revoke-keys/0.log"
Oct 30 05:38:22.270831 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:22.270804    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fa303d24-x7c2x_6428ca6d-5a07-4e1a-a99b-428054b598f7/revoke-keys/0.log"
Oct 30 05:38:22.467860 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:22.467832    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fc584f05-75t2b_62ef5729-9da4-4869-a370-e58c003c9916/revoke-keys/0.log"
Oct 30 05:38:22.655629 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:22.655555    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fdf757ae-nj74c_321853a1-5de7-4151-9c0c-abe5597c64a7/revoke-keys/0.log"
Oct 30 05:38:24.032737 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:24.032703    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"]
Oct 30 05:38:24.045054 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:24.045027    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff93de4-pf8dj"]
Oct 30 05:38:24.407629 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:24.407554    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4e97a8-c2e7-4922-9b04-42ef3182518b" path="/var/lib/kubelet/pods/da4e97a8-c2e7-4922-9b04-42ef3182518b/volumes"
Oct 30 05:38:26.018375 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:26.018337    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mjlc5_b85dd801-9b87-496b-b22a-ca557d3fd2c4/manager/0.log"
Oct 30 05:38:27.377696 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:27.377673    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-685cdf9dcb-c2k7g_ce064774-9d8f-4def-a363-809aae76d1ef/kube-auth-proxy/0.log"
Oct 30 05:38:28.180319 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:28.180292    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-gl57g_8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0/istio-proxy/0.log"
Oct 30 05:38:29.224271 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:29.224244    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-smfn4_ac611251-3890-4b60-b3a3-fb444a6befbc/istio-proxy/0.log"
Oct 30 05:38:29.437451 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:29.437422    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-txlk4_b748c4e2-2e95-43e6-8dc4-9bf3958a24d2/istio-proxy/0.log"
Oct 30 05:38:30.385365 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:30.385337    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84bd8c5fd7-885hp_b752c121-5e44-4df2-9f17-97f0c899516b/router/0.log"
Oct 30 05:38:34.030828 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:34.030792    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"]
Oct 30 05:38:34.043563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:34.043534    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b66617-q5cb2"]
Oct 30 05:38:34.407962 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:34.407884    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6cc501-417e-4a72-9331-b8ea9354f2b2" path="/var/lib/kubelet/pods/2b6cc501-417e-4a72-9331-b8ea9354f2b2/volumes"
Oct 30 05:38:41.752852 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:38:41.752812    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:38:43.552359 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.552315    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kkfx8/must-gather-wm4gv"]
Oct 30 05:38:43.553067 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.553046    5072 cpu_manager.go:414] "containerMap: removing container" podUID="62ef5729-9da4-4869-a370-e58c003c9916" containerName="revoke-keys"
Oct 30 05:38:43.553194 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.553071    5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ef5729-9da4-4869-a370-e58c003c9916" containerName="revoke-keys"
Oct 30 05:38:43.553278 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.553264    5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ef5729-9da4-4869-a370-e58c003c9916" containerName="revoke-keys"
Oct 30 05:38:43.557997 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.557974    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kkfx8/must-gather-wm4gv"
Oct 30 05:38:43.583349 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.583312    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kkfx8/must-gather-wm4gv"]
Oct 30 05:38:43.587861 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.587826    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kkfx8\"/\"kube-root-ca.crt\""
Oct 30 05:38:43.588050 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.587939    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kkfx8\"/\"default-dockercfg-vhbnz\""
Oct 30 05:38:43.588050 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.588020    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kkfx8\"/\"openshift-service-ca.crt\""
Oct 30 05:38:43.678819 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.678774    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ec55d70e-bd40-49ab-8ca7-962f13b4675b-must-gather-output\") pod \"must-gather-wm4gv\" (UID: \"ec55d70e-bd40-49ab-8ca7-962f13b4675b\") " pod="openshift-must-gather-kkfx8/must-gather-wm4gv"
Oct 30 05:38:43.679004 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.678831    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svd2\" (UniqueName: \"kubernetes.io/projected/ec55d70e-bd40-49ab-8ca7-962f13b4675b-kube-api-access-7svd2\") pod \"must-gather-wm4gv\" (UID: \"ec55d70e-bd40-49ab-8ca7-962f13b4675b\") " pod="openshift-must-gather-kkfx8/must-gather-wm4gv"
Oct 30 05:38:43.780389 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.780350    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ec55d70e-bd40-49ab-8ca7-962f13b4675b-must-gather-output\") pod \"must-gather-wm4gv\" (UID: \"ec55d70e-bd40-49ab-8ca7-962f13b4675b\") " pod="openshift-must-gather-kkfx8/must-gather-wm4gv"
Oct 30 05:38:43.780585 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.780406    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svd2\" (UniqueName: \"kubernetes.io/projected/ec55d70e-bd40-49ab-8ca7-962f13b4675b-kube-api-access-7svd2\") pod \"must-gather-wm4gv\" (UID: \"ec55d70e-bd40-49ab-8ca7-962f13b4675b\") " pod="openshift-must-gather-kkfx8/must-gather-wm4gv"
Oct 30 05:38:43.780802 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.780777    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ec55d70e-bd40-49ab-8ca7-962f13b4675b-must-gather-output\") pod \"must-gather-wm4gv\" (UID: \"ec55d70e-bd40-49ab-8ca7-962f13b4675b\") " pod="openshift-must-gather-kkfx8/must-gather-wm4gv"
Oct 30 05:38:43.804756 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.804678    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svd2\" (UniqueName: \"kubernetes.io/projected/ec55d70e-bd40-49ab-8ca7-962f13b4675b-kube-api-access-7svd2\") pod \"must-gather-wm4gv\" (UID: \"ec55d70e-bd40-49ab-8ca7-962f13b4675b\") " pod="openshift-must-gather-kkfx8/must-gather-wm4gv"
Oct 30 05:38:43.873493 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:43.873453    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kkfx8/must-gather-wm4gv"
Oct 30 05:38:44.181390 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:44.181360    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kkfx8/must-gather-wm4gv"]
Oct 30 05:38:44.182856 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:38:44.182829    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec55d70e_bd40_49ab_8ca7_962f13b4675b.slice/crio-fe453979182701eae9130f53e548f5611a425202b357355796b17ec7ac76594b WatchSource:0}: Error finding container fe453979182701eae9130f53e548f5611a425202b357355796b17ec7ac76594b: Status 404 returned error can't find the container with id fe453979182701eae9130f53e548f5611a425202b357355796b17ec7ac76594b
Oct 30 05:38:44.185251 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:44.185231    5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:38:44.609133 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:44.609096    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kkfx8/must-gather-wm4gv" event={"ID":"ec55d70e-bd40-49ab-8ca7-962f13b4675b","Type":"ContainerStarted","Data":"fe453979182701eae9130f53e548f5611a425202b357355796b17ec7ac76594b"}
Oct 30 05:38:47.915107 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:47.915041    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d67gd_bc9a426a-587d-4611-9298-90fc4b4f1dc3/global-pull-secret-syncer/0.log"
Oct 30 05:38:48.251381 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:48.251350    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7tc9g_d18cc860-7cd8-48d9-9a7a-40a40a53f379/connection-checker/0.log"
Oct 30 05:38:48.378196 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:48.378166    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-drvrv_5e68d10c-d217-40d3-945d-c87ccbd7434b/connection-checker/0.log"
Oct 30 05:38:48.494782 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:48.494750    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jmlk2_4455e2df-ec71-4748-8920-36b2374af2a1/connection-checker/0.log"
Oct 30 05:38:48.845651 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:48.845622    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d7nsr_8e7b0fed-186b-4d7b-affa-e91d5ef4b523/konnectivity-agent/0.log"
Oct 30 05:38:49.476936 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:49.476907    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-99.ec2.internal_f95db1d51f36d0a32fb67f82ccd47c76/haproxy/0.log"
Oct 30 05:38:49.656466 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:49.656284    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kkfx8/must-gather-wm4gv" event={"ID":"ec55d70e-bd40-49ab-8ca7-962f13b4675b","Type":"ContainerStarted","Data":"af5d2f11b11be0fd2aa5b99e9815e223ca97d559e136d133faba74697edb8024"}
Oct 30 05:38:50.664174 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:50.664137    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kkfx8/must-gather-wm4gv" event={"ID":"ec55d70e-bd40-49ab-8ca7-962f13b4675b","Type":"ContainerStarted","Data":"9b6ce90b7aa7896240e2dc710b7702c017c76a6d0c076e7cea6558f6615e0e00"}
Oct 30 05:38:50.708205 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:50.708149    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kkfx8/must-gather-wm4gv" podStartSLOduration=2.410101688 podStartE2EDuration="7.70813479s" totalImagesPullingTime="5.298033102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:50.704925289 +0000 UTC m=+6110.446348396" watchObservedRunningTime="2026-10-30 05:38:50.70813479 +0000 UTC m=+6110.449557886"
Oct 30 05:38:51.049777 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:51.049750    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6fxf_a80b6be8-ec4e-40c6-8889-256a744263b3/manager/0.log"
Oct 30 05:38:51.170924 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:51.170867    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qgl9w_a3f48faf-a0f6-4a0a-ac98-963183990dab/manager/0.log"
Oct 30 05:38:51.514167 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:51.514134    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6fxf_a80b6be8-ec4e-40c6-8889-256a744263b3/manager/0.log"
Oct 30 05:38:51.673239 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:51.673203    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qgl9w_a3f48faf-a0f6-4a0a-ac98-963183990dab/manager/0.log"
Oct 30 05:38:51.885897 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:51.885787    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-k6qmc_6671eae0-dd1d-41f5-b479-2c8e36e14650/manager/0.log"
Oct 30 05:38:52.091276 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:52.091247    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g9hm4_812448a2-1e32-4f19-86f0-8cf17f32edd1/server/0.log"
Oct 30 05:38:52.374836 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:52.374792    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-k6qmc_6671eae0-dd1d-41f5-b479-2c8e36e14650/manager/0.log"
Oct 30 05:38:52.736067 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:52.736041    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g9hm4_812448a2-1e32-4f19-86f0-8cf17f32edd1/server/0.log"
Oct 30 05:38:54.200608 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:54.200561    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-mb5bq_c7bec9d6-a8f4-4051-867a-b28db0efb1ed/proxy/0.log"
Oct 30 05:38:54.770778 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:54.770740    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-mb5bq_c7bec9d6-a8f4-4051-867a-b28db0efb1ed/proxy/0.log"
Oct 30 05:38:55.762266 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:55.762237    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6fxf_a80b6be8-ec4e-40c6-8889-256a744263b3/manager/0.log"
Oct 30 05:38:55.893660 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:55.893631    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qgl9w_a3f48faf-a0f6-4a0a-ac98-963183990dab/manager/0.log"
Oct 30 05:38:56.108855 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:56.108757    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6fxf_a80b6be8-ec4e-40c6-8889-256a744263b3/manager/0.log"
Oct 30 05:38:56.262495 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:56.262454    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qgl9w_a3f48faf-a0f6-4a0a-ac98-963183990dab/manager/0.log"
Oct 30 05:38:56.891062 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:56.891022    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-k6qmc_6671eae0-dd1d-41f5-b479-2c8e36e14650/manager/0.log"
Oct 30 05:38:57.038225 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:57.038188    5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"]
Oct 30 05:38:57.050400 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:57.050347    5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470c2cbb-5wznj"]
Oct 30 05:38:57.058806 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:57.058783    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-k6qmc_6671eae0-dd1d-41f5-b479-2c8e36e14650/manager/0.log"
Oct 30 05:38:57.631480 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:57.631443    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g9hm4_812448a2-1e32-4f19-86f0-8cf17f32edd1/server/0.log"
Oct 30 05:38:57.643679 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:57.643654    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g9hm4_812448a2-1e32-4f19-86f0-8cf17f32edd1/server/0.log"
Oct 30 05:38:58.408352 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:58.408313    5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a0f085-bff2-442a-8af2-6bcf36d3f47f" path="/var/lib/kubelet/pods/c2a0f085-bff2-442a-8af2-6bcf36d3f47f/volumes"
Oct 30 05:38:58.913018 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:58.912980    5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"]
Oct 30 05:38:58.920970 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:58.920930    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:58.934999 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:58.934969    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"]
Oct 30 05:38:59.022597 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.022560    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xfkr\"/\"openshift-service-ca.crt\""
Oct 30 05:38:59.022597 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.022573    5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9xfkr\"/\"default-dockercfg-x74pv\""
Oct 30 05:38:59.022597 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.022594    5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xfkr\"/\"kube-root-ca.crt\""
Oct 30 05:38:59.036285 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.036251    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ed467fc5-e803-44ec-9388-2514bc35d35f-proc\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.036520 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.036291    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed467fc5-e803-44ec-9388-2514bc35d35f-lib-modules\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.036520 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.036321    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ed467fc5-e803-44ec-9388-2514bc35d35f-podres\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.036520 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.036468    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq66k\" (UniqueName: \"kubernetes.io/projected/ed467fc5-e803-44ec-9388-2514bc35d35f-kube-api-access-pq66k\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.036520 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.036500    5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed467fc5-e803-44ec-9388-2514bc35d35f-sys\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.138152 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.138103    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ed467fc5-e803-44ec-9388-2514bc35d35f-proc\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.138152 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.138153    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed467fc5-e803-44ec-9388-2514bc35d35f-lib-modules\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.138396 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.138193    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ed467fc5-e803-44ec-9388-2514bc35d35f-podres\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.138396 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.138225    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq66k\" (UniqueName: \"kubernetes.io/projected/ed467fc5-e803-44ec-9388-2514bc35d35f-kube-api-access-pq66k\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.138396 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.138227    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ed467fc5-e803-44ec-9388-2514bc35d35f-proc\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.138396 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.138271    5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed467fc5-e803-44ec-9388-2514bc35d35f-sys\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.138396 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.138320    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ed467fc5-e803-44ec-9388-2514bc35d35f-podres\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.138396 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.138329    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed467fc5-e803-44ec-9388-2514bc35d35f-lib-modules\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.138703 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.138380    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed467fc5-e803-44ec-9388-2514bc35d35f-sys\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.163209 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.163137    5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq66k\" (UniqueName: \"kubernetes.io/projected/ed467fc5-e803-44ec-9388-2514bc35d35f-kube-api-access-pq66k\") pod \"perf-node-gather-daemonset-j7dd7\" (UID: \"ed467fc5-e803-44ec-9388-2514bc35d35f\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.238023 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.237993    5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.271549 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.271518    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mjlc5_b85dd801-9b87-496b-b22a-ca557d3fd2c4/manager/0.log"
Oct 30 05:38:59.556563 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.556536    5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"]
Oct 30 05:38:59.557891 ip-10-0-139-99 kubenswrapper[5072]: W1030 05:38:59.557863    5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poded467fc5_e803_44ec_9388_2514bc35d35f.slice/crio-b457d254dd1246f78468192130b5c03dd9e1e4978f57ec931e43d7d9f0014ced WatchSource:0}: Error finding container b457d254dd1246f78468192130b5c03dd9e1e4978f57ec931e43d7d9f0014ced: Status 404 returned error can't find the container with id b457d254dd1246f78468192130b5c03dd9e1e4978f57ec931e43d7d9f0014ced
Oct 30 05:38:59.727305 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.727264    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7" event={"ID":"ed467fc5-e803-44ec-9388-2514bc35d35f","Type":"ContainerStarted","Data":"ffd3462914346a90f89bec7fad5def0403c4c9b57daa12ee31edfa9697dcd04d"}
Oct 30 05:38:59.727562 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.727323    5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7" event={"ID":"ed467fc5-e803-44ec-9388-2514bc35d35f","Type":"ContainerStarted","Data":"b457d254dd1246f78468192130b5c03dd9e1e4978f57ec931e43d7d9f0014ced"}
Oct 30 05:38:59.727562 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.727481    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:38:59.771689 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:38:59.771635    5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7" podStartSLOduration=1.771620506 podStartE2EDuration="1.771620506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:59.770419565 +0000 UTC m=+6119.511842663" watchObservedRunningTime="2026-10-30 05:38:59.771620506 +0000 UTC m=+6119.513043603"
Oct 30 05:39:00.101031 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:00.101002    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-mb5bq_c7bec9d6-a8f4-4051-867a-b28db0efb1ed/proxy/0.log"
Oct 30 05:39:00.123336 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:00.123306    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-mb5bq_c7bec9d6-a8f4-4051-867a-b28db0efb1ed/proxy/0.log"
Oct 30 05:39:01.499799 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:01.499762    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-sb794_503b82e1-e56c-4c80-9d63-fa8e647c0991/cluster-monitoring-operator/0.log"
Oct 30 05:39:01.516146 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:01.516101    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6fxf_a80b6be8-ec4e-40c6-8889-256a744263b3/manager/0.log"
Oct 30 05:39:01.516146 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:01.516136    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6fxf_a80b6be8-ec4e-40c6-8889-256a744263b3/manager/0.log"
Oct 30 05:39:01.622314 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:01.622284    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-kst59_3a90b92e-569c-4908-b5f4-0b6643ae603b/kube-state-metrics/0.log"
Oct 30 05:39:01.679893 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:01.679858    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qgl9w_a3f48faf-a0f6-4a0a-ac98-963183990dab/manager/0.log"
Oct 30 05:39:01.681815 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:01.681791    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-qgl9w_a3f48faf-a0f6-4a0a-ac98-963183990dab/manager/0.log"
Oct 30 05:39:01.744962 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:01.744931    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-kst59_3a90b92e-569c-4908-b5f4-0b6643ae603b/kube-rbac-proxy-main/0.log"
Oct 30 05:39:01.878609 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:01.878533    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-kst59_3a90b92e-569c-4908-b5f4-0b6643ae603b/kube-rbac-proxy-self/0.log"
Oct 30 05:39:02.021247 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:02.021218    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7858fcf8fc-qc6gr_ed7968f8-8483-44d3-bd36-201b7f0bde51/metrics-server/0.log"
Oct 30 05:39:02.162745 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:02.162671    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-hkgg9_750f1340-7214-425f-8511-84fa94adb834/monitoring-plugin/0.log"
Oct 30 05:39:02.699909 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:02.699855    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-k6qmc_6671eae0-dd1d-41f5-b479-2c8e36e14650/manager/0.log"
Oct 30 05:39:02.700444 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:02.700024    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-k6qmc_6671eae0-dd1d-41f5-b479-2c8e36e14650/manager/0.log"
Oct 30 05:39:02.996814 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:02.996743    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g9hm4_812448a2-1e32-4f19-86f0-8cf17f32edd1/server/0.log"
Oct 30 05:39:02.998527 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:02.998504    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g9hm4_812448a2-1e32-4f19-86f0-8cf17f32edd1/server/0.log"
Oct 30 05:39:03.093124 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:03.093101    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h78rv_c61d535d-a6ff-4a10-84af-9c697e073ddb/node-exporter/0.log"
Oct 30 05:39:03.223522 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:03.223499    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h78rv_c61d535d-a6ff-4a10-84af-9c697e073ddb/kube-rbac-proxy/0.log"
Oct 30 05:39:03.356178 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:03.356102    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h78rv_c61d535d-a6ff-4a10-84af-9c697e073ddb/init-textfile/0.log"
Oct 30 05:39:03.488985 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:03.488960    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wmrjs_ebb63468-4a79-401b-ada4-5b2e1ac38197/kube-rbac-proxy-main/0.log"
Oct 30 05:39:03.618679 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:03.618598    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wmrjs_ebb63468-4a79-401b-ada4-5b2e1ac38197/kube-rbac-proxy-self/0.log"
Oct 30 05:39:03.748177 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:03.748147    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wmrjs_ebb63468-4a79-401b-ada4-5b2e1ac38197/openshift-state-metrics/0.log"
Oct 30 05:39:04.757174 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:04.757140    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-6vzvm_271ee094-c03d-4a43-b81d-d11b7262ff36/prometheus-operator/0.log"
Oct 30 05:39:04.886707 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:04.886678    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-6vzvm_271ee094-c03d-4a43-b81d-d11b7262ff36/kube-rbac-proxy/0.log"
Oct 30 05:39:05.019442 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:05.019368    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lwsgv_4e25fafd-0ea2-41d2-a9cf-2a8b1c9fa6b6/prometheus-operator-admission-webhook/0.log"
Oct 30 05:39:05.155781 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:05.155750    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78d6d46cf9-8zhdq_b142444b-c75c-480d-83bf-54350d59439c/telemeter-client/0.log"
Oct 30 05:39:05.287102 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:05.287026    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78d6d46cf9-8zhdq_b142444b-c75c-480d-83bf-54350d59439c/reload/0.log"
Oct 30 05:39:05.418536 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:05.418513    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78d6d46cf9-8zhdq_b142444b-c75c-480d-83bf-54350d59439c/kube-rbac-proxy/0.log"
Oct 30 05:39:05.558553 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:05.558484    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6948d6f8d4-4vfrk_c86a91cd-fdf5-4f67-a404-6011230b0123/thanos-query/0.log"
Oct 30 05:39:05.687730 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:05.687685    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6948d6f8d4-4vfrk_c86a91cd-fdf5-4f67-a404-6011230b0123/kube-rbac-proxy-web/0.log"
Oct 30 05:39:05.749335 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:05.749309    5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-j7dd7"
Oct 30 05:39:05.750735 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:05.750715    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-mb5bq_c7bec9d6-a8f4-4051-867a-b28db0efb1ed/proxy/0.log"
Oct 30 05:39:05.786340 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:05.786294    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-mb5bq_c7bec9d6-a8f4-4051-867a-b28db0efb1ed/proxy/0.log"
Oct 30 05:39:05.817602 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:05.817513    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6948d6f8d4-4vfrk_c86a91cd-fdf5-4f67-a404-6011230b0123/kube-rbac-proxy/0.log"
Oct 30 05:39:05.946956 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:05.946925    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6948d6f8d4-4vfrk_c86a91cd-fdf5-4f67-a404-6011230b0123/prom-label-proxy/0.log"
Oct 30 05:39:06.076842 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:06.076761    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6948d6f8d4-4vfrk_c86a91cd-fdf5-4f67-a404-6011230b0123/kube-rbac-proxy-rules/0.log"
Oct 30 05:39:06.205768 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:06.205740    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6948d6f8d4-4vfrk_c86a91cd-fdf5-4f67-a404-6011230b0123/kube-rbac-proxy-metrics/0.log"
Oct 30 05:39:07.497917 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:07.497886    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-685cdf9dcb-c2k7g_ce064774-9d8f-4def-a363-809aae76d1ef/kube-auth-proxy/0.log"
Oct 30 05:39:07.586491 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:07.586464    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-685cdf9dcb-c2k7g_ce064774-9d8f-4def-a363-809aae76d1ef/kube-auth-proxy/0.log"
Oct 30 05:39:07.759317 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:07.759243    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-htskn_01bc8bd9-f593-4acf-97c0-fc6c4beb4867/networking-console-plugin/0.log"
Oct 30 05:39:07.760895 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:07.760877    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-gl57g_8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0/istio-proxy/0.log"
Oct 30 05:39:07.846867 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:07.846833    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-gl57g_8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0/istio-proxy/0.log"
Oct 30 05:39:08.083836 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:08.083722    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-smfn4_ac611251-3890-4b60-b3a3-fb444a6befbc/istio-proxy/0.log"
Oct 30 05:39:08.148259 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:08.148219    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-txlk4_b748c4e2-2e95-43e6-8dc4-9bf3958a24d2/istio-proxy/0.log"
Oct 30 05:39:08.171918 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:08.171893    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-smfn4_ac611251-3890-4b60-b3a3-fb444a6befbc/istio-proxy/0.log"
Oct 30 05:39:08.240765 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:08.240741    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-txlk4_b748c4e2-2e95-43e6-8dc4-9bf3958a24d2/istio-proxy/0.log"
Oct 30 05:39:08.518896 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:08.518868    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84bd8c5fd7-885hp_b752c121-5e44-4df2-9f17-97f0c899516b/router/0.log"
Oct 30 05:39:08.591456 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:08.591400    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84bd8c5fd7-885hp_b752c121-5e44-4df2-9f17-97f0c899516b/router/0.log"
Oct 30 05:39:09.010464 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:09.010412    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/1.log"
Oct 30 05:39:09.014328 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:09.014297    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-664tl_907912b8-9313-43b3-b324-e8498ea6519a/console-operator/2.log"
Oct 30 05:39:10.251549 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:10.251516    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-j55cl_5bb6afde-1ba7-4045-841d-528387cf7f9d/download-server/0.log"
Oct 30 05:39:11.462650 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:11.462623    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-cp4k2_4d0cc1cb-a036-4ddb-92c1-8a900fc7bc4b/volume-data-source-validator/0.log"
Oct 30 05:39:11.751889 ip-10-0-139-99 kubenswrapper[5072]: E1030 05:39:11.751808    5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7x92g" containerName="extractor"
Oct 30 05:39:12.439433 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:12.439404    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-685cdf9dcb-c2k7g_ce064774-9d8f-4def-a363-809aae76d1ef/kube-auth-proxy/0.log"
Oct 30 05:39:12.727512 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:12.727430    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-gl57g_8f2b57e1-69fa-4d2d-a8f9-980e5dfb3dd0/istio-proxy/0.log"
Oct 30 05:39:13.119299 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:13.119220    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-smfn4_ac611251-3890-4b60-b3a3-fb444a6befbc/istio-proxy/0.log"
Oct 30 05:39:13.192859 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:13.192825    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-txlk4_b748c4e2-2e95-43e6-8dc4-9bf3958a24d2/istio-proxy/0.log"
Oct 30 05:39:13.555530 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:13.555500    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84bd8c5fd7-885hp_b752c121-5e44-4df2-9f17-97f0c899516b/router/0.log"
Oct 30 05:39:14.768120 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:14.768068    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xkq6m_730fcc37-b18e-46e3-aea5-a7fba60b7a8c/dns/0.log"
Oct 30 05:39:14.901947 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:14.901881    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xkq6m_730fcc37-b18e-46e3-aea5-a7fba60b7a8c/kube-rbac-proxy/0.log"
Oct 30 05:39:15.330037 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:15.330006    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6njd4_b555cf82-5423-4c47-a004-ae399654683b/dns-node-resolver/0.log"
Oct 30 05:39:15.387957 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:15.387899    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mjlc5_b85dd801-9b87-496b-b22a-ca557d3fd2c4/manager/0.log"
Oct 30 05:39:15.530351 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:15.530306    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mjlc5_b85dd801-9b87-496b-b22a-ca557d3fd2c4/manager/0.log"
Oct 30 05:39:16.936047 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:16.936014    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6878899c59-ksp2c_243e61df-e18d-402e-bef1-f2713f198a08/registry/0.log"
Oct 30 05:39:17.218844 ip-10-0-139-99 kubenswrapper[5072]: I1030 05:39:17.218725    5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mc77t_0384e218-1937-4099-8566-67775cda0521/node-ca/0.log"